Coverage for cc_modules/cc_cache.py: 100%

1# noinspection HttpUrlsUsage

2"""

3 camcops_server/cc_modules/cc_cache.py

4

5===============================================================================

6

7 Copyright (C) 2012, University of Cambridge, Department of Psychiatry.

8 Created by Rudolf Cardinal (rnc1001@cam.ac.uk).

9

10 This file is part of CamCOPS.

11

12 CamCOPS is free software: you can redistribute it and/or modify

13 it under the terms of the GNU General Public License as published by

14 the Free Software Foundation, either version 3 of the License, or

15 (at your option) any later version.

16

17 CamCOPS is distributed in the hope that it will be useful,

18 but WITHOUT ANY WARRANTY; without even the implied warranty of

19 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

20 GNU General Public License for more details.

21

22 You should have received a copy of the GNU General Public License

23 along with CamCOPS. If not, see <https://www.gnu.org/licenses/>.

24

25===============================================================================

26

27**Cache functions.**

28

29 1. The basic cache objects.

30

31 2. FIX FOR DOGPILE.CACHE FOR DECORATED FUNCTIONS, 2017-07-28 (PLUS SOME OTHER

32 IMPROVEMENTS). SEE

33 https://bitbucket.org/zzzeek/dogpile.cache/issues/96/error-in-python-35-with-use-of-deprecated

34

35 Crash using type-hinted functions under Python 3.5 with dogpile.cache==0.6.4:

36

37.. code-block:: none

38

39 Traceback (most recent call last):

40 File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main

41 "__main__", mod_spec)

42 File "/usr/lib/python3.5/runpy.py", line 85, in _run_code

43 exec(code, run_globals)

44 File "/home/rudolf/Documents/code/camcops/server/camcops_server/cc_modules/cc_cache.py", line 64, in <module>

45 unit_test_cache()

46 File "/home/rudolf/Documents/code/camcops/server/camcops_server/cc_modules/cc_cache.py", line 50, in unit_test_cache

47 def testfunc() -> str:

48 File "/home/rudolf/dev/venvs/camcops/lib/python3.5/site-packages/dogpile/cache/region.py", line 1215, in decorator

49 key_generator = function_key_generator(namespace, fn)

50 File "/home/rudolf/dev/venvs/camcops/lib/python3.5/site-packages/dogpile/cache/util.py", line 31, in function_key_generator

51 args = inspect.getargspec(fn)

52 File "/usr/lib/python3.5/inspect.py", line 1045, in getargspec

53 raise ValueError("Function has keyword-only arguments or annotations"

54 ValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them

55

56 3. CACHING NOTES

57

58- We currently use 'dogpile.cache.memory' as the backend.

59 This means that for single-process (single-thread or multithreaded) servers,

60 the cache is unique, but that won't work for multi-process (e.g. Gunicorn)

61 servers.

62

63- That means that in a multiprocess environment it's fine to continue to use a

64 memory cache for file-based stuff (e.g. configs, XML strings), but not for

65 database-based stuff (e.g. which ID numbers are valid).

66

67- Correct solutions WITH a cache for those database-based things include:

68

69 - ignoring Python caching and relying on the MySQL query cache -- but this is

70 being removed because it's not all that great:

71

72 https://mysqlserverteam.com/mysql-8-0-retiring-support-for-the-query-cache/

73

74 - using memcached (via dogpile.cache.pylibmc)

75

76 https://www.ubergizmo.com/how-to/install-memcached-windows/

77

78 - using redis (via dogpile.cache.redis and

79 https://pypi.python.org/pypi/redis/ )

80

81 https://stackoverflow.com/questions/10558465/memcached-vs-redis

82 https://redis.io/

83 https://web.archive.org/web/20120118030804/http://simonwillison.net/static/2010/redis-tutorial/

84 http://oldblog.antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html

85 https://redis.io/topics/security

86

87 redis unsupported under Windows:

88 https://redis.io/download

89

90- The other obvious alternative: don't cache such stuff! This may all be

91 premature optimization.

92

93 https://msol.io/blog/tech/youre-probably-wrong-about-caching/

94

95 The actual price is of the order of 0.6-1 ms per query, for the queries

96 "find me all the ID number definitions" and "fetch the server settings".

97

98- The answer is probably:

99

100 - continue to use dogpile.cache.memory for simple "fixed" stuff read from

101 disk;

102 - continue to use Pyramid's per-request caching mechanism (@reify);

103 - forget about database caching for now;

104 - if it becomes a problem later, move to Redis

105

106- Therefore:

107

108 - there should be no calls to cache_region_static.delete

109

110""" # noqa

111

112

113# =============================================================================

114# Imports; logging

115# =============================================================================

116

117 from cardinal_pythonlib.dogpile_cache import kw_fkg_allowing_type_hints as fkg

118 from dogpile.cache import make_region

119

120

121# =============================================================================

122# The main cache: static for the lifetime of this process.

123# =============================================================================

124

125 cache_region_static = make_region()

126 cache_region_static.configure(

127 backend="dogpile.cache.memory"

128 # Consider later: memcached via dogpile.cache.pylibmc

129)

130

131# Can now use:

132# @cache_region_static.cache_on_arguments(function_key_generator=fkg)

133

134# https://stackoverflow.com/questions/44834/can-someone-explain-all-in-python

135 __all__ = ["cache_region_static", "fkg"] # prevents "Unused import statement"