Coverage for cc_modules/client_api.py: 91%

1"""

2 camcops_server/cc_modules/client_api.py

3

4===============================================================================

5

6 Copyright (C) 2012, University of Cambridge, Department of Psychiatry.

7 Created by Rudolf Cardinal (rnc1001@cam.ac.uk).

8

9 This file is part of CamCOPS.

10

11 CamCOPS is free software: you can redistribute it and/or modify

12 it under the terms of the GNU General Public License as published by

13 the Free Software Foundation, either version 3 of the License, or

14 (at your option) any later version.

15

16 CamCOPS is distributed in the hope that it will be useful,

17 but WITHOUT ANY WARRANTY; without even the implied warranty of

18 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

19 GNU General Public License for more details.

20

21 You should have received a copy of the GNU General Public License

22 along with CamCOPS. If not, see <https://www.gnu.org/licenses/>.

23

24===============================================================================

25

26**Implements the API through which client devices (tablets etc.) upload and

27 download data.**

28

29 We use primarily SQLAlchemy Core here (in contrast to the ORM used elsewhere).

30

31 This code is optimized to a degree for speed over clarity, aiming primarily to

32 reduce the number of database hits.

33

34**The overall upload method is as follows**

35

36 Everything that follows refers to records relating to a specific client device

37 in the "current" era, only.

38

39 In the preamble, the client:

40

41- verifies authorization via :func:`op_check_device_registered` and

42 :func:`op_check_upload_user_and_device`;

43- fetches and checks server ID information via :func:`op_get_id_info`;

44- checks its patients are acceptable via :func:`op_validate_patients`;

45- checks which tables are permitted via :func:`op_get_allowed_tables`;

46- performs some internal validity checks.

47

48 Then, in the usual stepwise upload:

49

50- :func:`op_start_upload`

51

52 - Rolls back any previous incomplete changes via :func:`rollback_all`.

53 - Creates an upload batch, via :func:`get_batch_details_start_if_needed`.

54

55- If were are in a preserving/finalizing upload: :func:`op_start_preservation`.

56

57 - Marks all tables as dirty.

58 - Marks the upload batch as a "preserving" batch.

59

60- Then call some or all of:

61

62 - For tables that are empty on the client, :func:`op_upload_empty_tables`.

63

64 - Current client records are marked as ``_removal_pending``.

65 - Any table that had previous client records is marked as dirty.

66 - If preserving, any table without current records is marked as clean.

67

68 - For tables that the client wishes to send in one go,

69 :func:`op_upload_table`.

70

71 - Find current server records.

72 - Use :func:`upload_record_core` to add new records and modify existing

73 ones, and :func:`flag_deleted` to delete ones that weren't on the client.

74 - If any records are new, modified, or deleted, mark the table as dirty.

75 - If preserving and there were no server records in this table, mark the

76 table as clean.

77

78 - For tables (e.g. BLOBs) that might be too big to send in one go:

79

80 - client sends PKs to :func:`op_delete_where_key_not`, which "deletes" all

81 other records, via :func:`flag_deleted_where_clientpk_not`.

82 - client sends PK and timestamp values to :func:`op_which_keys_to_send`

83 - server "deletes" records that are not in the list (via

84 :func:`flag_deleted_where_clientpk_not`, which marks the table as dirty

85 if any records were thus modified). Note REDUNDANCY here re

86 :func:`op_delete_where_key_not`.

87 - server tells the client which records are new or need to be updated

88 - client sends each of those via :func:`op_upload_record`

89

90 - Calls :func`upload_record_core`.

91 - Marks the table as dirty, unless the client erroneously sent an

92 unchanged record.

93

94- In addition, specific records can be marked as ``_move_off_tablet``.

95

96 - :func:`upload_record_core` checks this for otherwise "identical" records

97 and applies that flag to the server.

98

99- When the client's finished, it calls :func:`op_end_upload`.

100

101 - Calls :func:`commit_all`;

102 - ... which, for all dirty tables, calls :func:`commit_table`;

103 - ... which executes the "add", "remove", and "preserve" functions for the

104 table;

105 - ... and triggers the updating of special server indexes on patient ID

106 numbers and tasks, via :func:`update_indexes`.

107 - At the end, :func:`commit_all` clears the dirty-table list.

108

109 There's a little bit of special code to handle old tablet software, too.

110

111 As of v2.3.0, the function :func:`op_upload_entire_database` does this in one

112 step (faster; for use if the network packets are not excessively large).

113

114- Code relating to this uses ``batchdetails.onestep``.

115

116**Setup for the upload code**

117

118- Fire up a CamCOPS client with an empty database, e.g. from the build

119 directory via

120

121 .. code-block:: bash

122

123 ./camcops --dbdir ~/tmp/camcops_client_test

124

125- Fire up a web browser showing both (a) the task list via the index, and (b)

126 the task list without using the index. We'll use this to verify correct

127 indexing. **The two versions of the view should never be different.**

128

129- Ensure the test client device has no current records (force-finalize if

130 required).

131

132- Ensure the server's index is proper. Run ``camcops_server reindex`` if

133 required.

134

135- If required, fire up MySQL with the server database. You may wish to use

136 ``pager less -SFX``, for better display of large tables.

137

138**Testing the upload code**

139

140 Perform the following steps both (1) with the client forced to the stepwise

141 upload method, and (2) with it forced to one-step upload.

142

143 Note that the number of patient ID numbers uploaded (etc.) is ignored below.

144

145*Single record*

146

147[Checked for one-step and multi-step upload, 2018-11-21.]

148

149#. Create a blank ReferrerSatisfactionSurvey (table ``ref_satis_gen``).

150 This has the advantage of being an anonymous single-record task.

151

152#. Upload/copy.

153

154 - The server log should show 1 × ref_satis_gen added.

155

156 - The task lists should show the task as current and incomplete.

157

158#. Modify it, so it's complete.

159

160#. Upload/copy.

161

162 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen

163 modified out.

164

165 - The task lists should show the task as current and complete.

166

167#. Upload/move.

168

169 - The server log should show 2 × ref_satis_gen preserved.

170

171 - The task lists should show the task as no longer current.

172

173#. Create another blank one.

174

175#. Upload/copy.

176

177#. Modify it so it's complete.

178

179#. Specifically flag it for preservation (the chequered flags).

180

181#. Upload/copy.

182

183 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen

184 modified out, 2 × ref_satis_gen preserved.

185

186 - The task lists should show the task as complete and no longer current.

187

188*With a patient*

189

190[Checked for one-step and multi-step upload, 2018-11-21.]

191

192#. Create a dummy patient that the server will accept.

193

194#. Create a Progress Note with location "loc1" and abort its creation, giving

195 an incomplete task.

196

197#. Create a second Progress Note with location "loc2" and contents "note2".

198

199#. Create a third Progress Note with location "loc3" and contents "note3".

200

201#. Upload/copy. Verify. This checks *addition*.

202

203 - The server log should show 1 × patient added; 3 × progressnote added.

204 (Also however many patientidnum records you chose.)

205 - All three tasks should be "current".

206 - The first should be "incomplete".

207

208#. Modify the first note by adding contents "note1".

209

210#. Delete the second note.

211

212#. Upload/copy. Verify. This checks *modification*, *deletion*,

213 *no-change detection*, and *reindexing*.

214

215 - The server log should show 1 × progressnote added, 1 × progressnote

216 modified out, 1 × progressnote deleted.

217 - The first note should now appear as complete.

218 - The second should have vanished.

219 - The third should be unchanged.

220 - The two remaining tasks should still be "current".

221

222#. Delete the contents from the first note again.

223

224#. Upload/move (or move-keeping-patients; that's only different on the

225 client side). Verify. This checks *preservation (finalizing)* and

226 *reindexing*.

227

228 - The server log should show 1 × patient preserved; 1 × progressnote added,

229 1 × progressnote modified out, 5 × progressnote preserved.

230 - The two remaining tasks should no longer be "current".

231 - The first should no longer be "complete".

232

233#. Create a complete "note 4" and an incomplete "note 5".

234

235#. Upload/copy.

236

237#. Force-finalize from the server. This tests force-finalizing including

238 reindexing.

239

240 - The "tasks to finalize" list should have just two tasks in it.

241 - After force-finalizing, the tasks should remain in the index but no

242 longer be marked as current.

243

244#. Upload/move to get rid of the residual tasks on the client.

245

246 - The server log should show 1 × patient added, 1 × patient preserved; 2 ×

247 progressnote added, 2 × progressnote preserved.

248

249*With ancillary tables and BLOBs*

250

251[Checked for one-step and multi-step upload, 2018-11-21.]

252

253#. Create a PhotoSequence with text "t1", one photo named "p1" of you holding

254 up one finger vertically, and another photo named "p2" of you holding up

255 two fingers vertically.

256

257#. Upload/copy.

258

259 - The server log should show:

260

261 - blobs: 2 × added;

262 - patient: 1 × added;

263 - photosequence: 1 × added;

264 - photosequence_photos: 2 × added.

265

266 - The task lists should look sensible.

267

268#. Clear the second photo and replace it with a photo of you holding up

269 two fingers horizontally.

270

271#. Upload/copy.

272

273 - The server log should show:

274

275 - blobs: 1 × added, 1 × modified out;

276 - photosequence: 1 × added, 1 × modified out;

277 - photosequence_photos: 1 × added, 1 × modified out.

278

279 - The task lists should look sensible.

280

281#. Back to two fingers vertically. (This is the fourth photo overall.)

282

283#. Mark that patient for specific finalization.

284

285#. Upload/copy.

286

287 - The server log should show:

288

289 - blobs: 1 × added, 1 × modified out, 4 × preserved;

290 - patient: 1 × preserved;

291 - photosequence: 1 × added, 1 × modified out, 3 × preserved;

292 - photosequence_photos: 1 × added, 1 × modified out, 4 × preserved.

293

294 - The tasks should no longer be current.

295 - A fresh "vertical fingers" photo should be visible.

296

297#. Create another patient and another PhotoSequence with one photo of three

298 fingers.

299

300#. Upload-copy.

301

302#. Force-finalize.

303

304 - Should finalize: 1 × blobs, 1 × patient, 1 × photosequence, 1 ×

305 photosequence_photos.

306

307#. Upload/move.

308

309 During any MySQL debugging, remember:

310

311.. code-block:: none

312

313 -- For better display:

314 pager less -SFX;

315

316 -- To view relevant parts of the BLOB table without the actual BLOB:

317

318 SELECT

319 _pk, _group_id, _device_id, _era,

320 _current, _predecessor_pk, _successor_pk,

321 _addition_pending, _when_added_batch_utc, _adding_user_id,

322 _removal_pending, _when_removed_batch_utc, _removing_user_id,

323 _move_off_tablet,

324 _preserving_user_id, _forcibly_preserved,

325 id, tablename, tablepk, fieldname, mimetype, when_last_modified

326 FROM blobs;

327

328"""

329

330# =============================================================================

331# Imports

332# =============================================================================

333

334 import logging

335 import json

336

337# from pprint import pformat

338 import secrets

339 import string

340 import time

341 from typing import (

342 Any,

343 Dict,

344 Iterable,

345 List,

346 Optional,

347 Sequence,

348 Set,

349 Tuple,

350 TYPE_CHECKING,

351)

352 from cardinal_pythonlib.datetimefunc import (

353 coerce_to_pendulum,

354 coerce_to_pendulum_date,

355 format_datetime,

356)

357 from cardinal_pythonlib.httpconst import HttpMethod

358 from cardinal_pythonlib.logs import BraceStyleAdapter

359 from cardinal_pythonlib.pyramid.responses import TextResponse

360 from cardinal_pythonlib.sqlalchemy.core_query import (

361 exists_in_table,

362 fetch_all_first_values,

363)

364 from cardinal_pythonlib.text import escape_newlines

365 from pendulum.exceptions import ParserError

366 from pyramid.httpexceptions import HTTPBadRequest

367 from pyramid.view import view_config

368 from pyramid.response import Response

369 from pyramid.security import NO_PERMISSION_REQUIRED

370 from semantic_version import Version

371 from sqlalchemy.engine import CursorResult

372 from sqlalchemy.exc import IntegrityError

373 from sqlalchemy.orm import joinedload

374 from sqlalchemy.sql.expression import exists, select, update

375 from sqlalchemy.sql.schema import Table

376

377 from camcops_server.cc_modules import cc_audit # avoids "audit" name clash

378 from camcops_server.cc_modules.cc_all_models import CLIENT_TABLE_MAP

379 from camcops_server.cc_modules.cc_blob import Blob

380 from camcops_server.cc_modules.cc_cache import cache_region_static, fkg

381 from camcops_server.cc_modules.cc_client_api_core import (

382 AllowedTablesFieldNames,

383 BatchDetails,

384 exception_description,

385 ExtraStringFieldNames,

386 fail_server_error,

387 fail_unsupported_operation,

388 fail_user_error,

389 get_server_live_records,

390 require_keys,

391 ServerErrorException,

392 ServerRecord,

393 TabletParam,

394 UploadRecordResult,

395 UploadTableChanges,

396 UserErrorException,

397 values_delete_later,

398 values_delete_now,

399 values_preserve_now,

400 WhichKeyToSendInfo,

401)

402 from camcops_server.cc_modules.cc_client_api_helpers import (

403 upload_commit_order_sorter,

404)

405 from camcops_server.cc_modules.cc_constants import (

406 CLIENT_DATE_FIELD,

407 DateFormat,

408 ERA_NOW,

409 FP_ID_NUM,

410 FP_ID_DESC,

411 FP_ID_SHORT_DESC,

412 MOVE_OFF_TABLET_FIELD,

413 NUMBER_OF_IDNUMS_DEFUNCT, # allowed; for old tablet versions

414 POSSIBLE_SEX_VALUES,

415 TABLET_ID_FIELD,

416)

417 from camcops_server.cc_modules.cc_convert import (

418 decode_single_value,

419 decode_values,

420 encode_single_value,

421)

422 from camcops_server.cc_modules.cc_db import (

423 FN_ADDING_USER_ID,

424 FN_ADDITION_PENDING,

425 FN_CAMCOPS_VERSION,

426 FN_CURRENT,

427 FN_DEVICE_ID,

428 FN_ERA,

429 FN_GROUP_ID,

430 FN_PK,

431 FN_PREDECESSOR_PK,

432 FN_REMOVAL_PENDING,

433 FN_REMOVING_USER_ID,

434 FN_SUCCESSOR_PK,

435 FN_WHEN_ADDED_BATCH_UTC,

436 FN_WHEN_ADDED_EXACT,

437 FN_WHEN_REMOVED_BATCH_UTC,

438 FN_WHEN_REMOVED_EXACT,

439 RESERVED_FIELDS,

440)

441 from camcops_server.cc_modules.cc_device import Device

442 from camcops_server.cc_modules.cc_dirtytables import DirtyTable

443 from camcops_server.cc_modules.cc_group import Group

444 from camcops_server.cc_modules.cc_ipuse import IpUse

445 from camcops_server.cc_modules.cc_membership import UserGroupMembership

446 from camcops_server.cc_modules.cc_patient import (

447 Patient,

448 is_candidate_patient_valid_for_group,

449 is_candidate_patient_valid_for_restricted_user,

450)

451 from camcops_server.cc_modules.cc_patientidnum import (

452 fake_tablet_id_for_patientidnum,

453 PatientIdNum,

454)

455 from camcops_server.cc_modules.cc_proquint import (

456 InvalidProquintException,

457 uuid_from_proquint,

458)

459 from camcops_server.cc_modules.cc_pyramid import Routes

460 from camcops_server.cc_modules.cc_simpleobjects import (

461 BarePatientInfo,

462 IdNumReference,

463)

464 from camcops_server.cc_modules.cc_specialnote import SpecialNote

465 from camcops_server.cc_modules.cc_task import (

466 all_task_tables_with_min_client_version,

467)

468 from camcops_server.cc_modules.cc_taskindex import (

469 update_indexes_and_push_exports,

470)

471 from camcops_server.cc_modules.cc_user import User

472 from camcops_server.cc_modules.cc_validators import (

473 STRING_VALIDATOR_TYPE,

474 validate_anything,

475 validate_email,

476)

477 from camcops_server.cc_modules.cc_version import (

478 CAMCOPS_SERVER_VERSION_STRING,

479 MINIMUM_TABLET_VERSION,

480)

481

482 if TYPE_CHECKING:

483 from camcops_server.cc_modules.cc_request import CamcopsRequest

484

485 log = BraceStyleAdapter(logging.getLogger(__name__))

486

487

488# =============================================================================

489# Constants

490# =============================================================================

491

492 DUPLICATE_FAILED = "Failed to duplicate record"

493 INSERT_FAILED = "Failed to insert record"

494

495 SUCCESS_MSG = "Success"

496 SUCCESS_CODE = "1"

497 FAILURE_CODE = "0"

498

499 DEBUG_UPLOAD = False

500

501

502# =============================================================================

503# Quasi-constants

504# =============================================================================

505

506 DB_JSON_DECODER = json.JSONDecoder() # just a plain one

507 PATIENT_INFO_JSON_DECODER = json.JSONDecoder() # just a plain one

508

509

510# =============================================================================

511# Cached information

512# =============================================================================

513

514

515@cache_region_static.cache_on_arguments(function_key_generator=fkg)

516 def all_tables_with_min_client_version() -> Dict[str, Version]:

517 """

518 For all tables that the client might upload to, return a mapping from the

519 table name to the corresponding minimum client version.

520 """

521 d = all_task_tables_with_min_client_version()

522 d[Blob.__tablename__] = MINIMUM_TABLET_VERSION

523 d[Patient.__tablename__] = MINIMUM_TABLET_VERSION

524 d[PatientIdNum.__tablename__] = MINIMUM_TABLET_VERSION

525 return d

526

527

528# =============================================================================

529# Validators

530# =============================================================================

531

532

533 def ensure_valid_table_name(req: "CamcopsRequest", tablename: str) -> None:

534 """

535 Ensures a table name:

536

537 - doesn't contain bad characters,

538 - isn't a reserved table that the user is prohibited from accessing, and

539 - is a valid table name that's in the database.

540

541 Raises :exc:`UserErrorException` upon failure.

542

543 - 2017-10-08: shortcut to all that: it's OK if it's listed as a valid

544 client table.

545 - 2018-01-16 (v2.2.0): check also that client version is OK

546 """

547 if tablename not in CLIENT_TABLE_MAP:

548 fail_user_error(f"Invalid client table name: {tablename}")

549 tables_versions = all_tables_with_min_client_version()

550 assert tablename in tables_versions

551 client_version = req.tabletsession.tablet_version_ver

552 minimum_client_version = tables_versions[tablename]

553 if client_version < minimum_client_version:

554 fail_user_error(

555 f"Client CamCOPS version {client_version} is less than the "

556 f"version ({minimum_client_version}) "

557 f"required to handle table {tablename}"

558 )

559

560

561 def ensure_valid_field_name(table: Table, fieldname: str) -> None:

562 """

563 Ensures a field name contains only valid characters, and isn't a

564 reserved fieldname that the user isn't allowed to access.

565

566 Raises :exc:`UserErrorException` upon failure.

567

568 - 2017-10-08: shortcut: it's OK if it's a column name for a particular

569 table.

570 """

571 if fieldname.startswith("_"): # all reserved fields start with _

572 # ... but not all fields starting with "_" are reserved; e.g.

573 # "_move_off_tablet" is allowed.

574 if fieldname in RESERVED_FIELDS:

575 fail_user_error(

576 f"Reserved field name for table {table.name!r}: {fieldname!r}"

577 )

578 if fieldname not in table.columns.keys():

579 fail_user_error(

580 f"Invalid field name for table {table.name!r}: {fieldname!r}"

581 )

582 # Note that the reserved-field check is case-sensitive, but so is the

583 # "present in table" check. So for a malicious uploader trying to use, for

584 # example, "_PK", this would not be picked up as a reserved field (so would

585 # pass that check) but then wouldn't be recognized as a valid field (so

586 # would fail).

587

588

589 def ensure_string(value: Any, allow_none: bool = True) -> None:

590 """

591 Used when processing JSON information about patients: ensures that a value

592 is a string, or raises.

593

594 Args:

595 value: value to test

596 allow_none: is ``None`` allowed (not just an empty string)?

597 """

598 if value is None:

599 if allow_none:

600 return # OK

601 else:

602 fail_user_error("Patient JSON contains absent string")

603 if not isinstance(value, str):

604 fail_user_error(f"Patient JSON contains invalid non-string: {value!r}")

605

606

607 def ensure_valid_patient_json(

608 req: "CamcopsRequest", group: Group, pt_dict: Dict[str, Any]

609) -> None:

610 """

611 Ensures that the JSON dictionary contains valid patient details (valid for

612 the group into which it's being uploaded), and that (if applicable) this

613 user is allowed to upload this patient.

614

615 Args:

616 req:

617 the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

618 group:

619 the :class:`camcops_server.cc_modules.cc_group.Group` into which

620 the upload is going

621 pt_dict:

622 a JSON dictionary from the client

623

624 Raises:

625 :exc:`UserErrorException` if invalid

626

627 """

628 if not isinstance(pt_dict, dict):

629 fail_user_error("Patient JSON is not a dict")

630 if not pt_dict:

631 fail_user_error("Patient JSON is empty")

632 valid_which_idnums = req.valid_which_idnums

633 errors = [] # type: List[str]

634 finalizing = None

635 ptinfo = BarePatientInfo()

636 idnum_types_seen = set() # type: Set[int]

637 for k, v in pt_dict.items():

638 # May not be necessary as JSON has already been validated

639 ensure_string(k, allow_none=False)

640

641 if k == TabletParam.FORENAME:

642 ensure_string(v)

643 ptinfo.forename = v

644

645 elif k == TabletParam.SURNAME:

646 ensure_string(v)

647 ptinfo.surname = v

648

649 elif k == TabletParam.SEX:

650 if v not in POSSIBLE_SEX_VALUES:

651 fail_user_error(f"Bad sex value: {v!r}")

652 ptinfo.sex = v

653

654 elif k == TabletParam.DOB:

655 ensure_string(v)

656 if v:

657 try:

658 # This will only return None if v is empty/None and we have

659 # already checked that

660 dob = coerce_to_pendulum_date(v)

661 except ParserError:

662 fail_user_error(f"Invalid DOB: {v!r}")

663 else:

664 dob = None

665 ptinfo.dob = dob

666

667 elif k == TabletParam.EMAIL:

668 ensure_string(v)

669 if v:

670 try:

671 validate_email(v)

672 except ValueError:

673 fail_user_error(f"Bad e-mail address: {v!r}")

674 ptinfo.email = v

675

676 elif k == TabletParam.ADDRESS:

677 ensure_string(v)

678 ptinfo.address = v

679

680 elif k == TabletParam.GP:

681 ensure_string(v)

682 ptinfo.gp = v

683

684 elif k == TabletParam.OTHER:

685 ensure_string(v)

686 ptinfo.otherdetails = v

687

688 elif k.startswith(TabletParam.IDNUM_PREFIX):

689 nstr = k[len(TabletParam.IDNUM_PREFIX) :]

690 try:

691 which_idnum = int(nstr)

692 except (TypeError, ValueError):

693 fail_user_error(f"Bad idnum key: {k!r}")

694 # noinspection PyUnboundLocalVariable

695 if which_idnum not in valid_which_idnums:

696 fail_user_error(f"Bad ID number type: {which_idnum}")

697 if which_idnum in idnum_types_seen:

698 fail_user_error(

699 f"More than one ID number supplied for ID "

700 f"number type {which_idnum}"

701 )

702 idnum_types_seen.add(which_idnum)

703 if v is not None and not isinstance(v, int):

704 fail_user_error(f"Bad ID number value: {v!r}")

705 idref = IdNumReference(which_idnum, v)

706 if not idref.is_valid():

707 fail_user_error(f"Bad ID number: {idref!r}")

708 ptinfo.add_idnum(idref)

709

710 elif k == TabletParam.FINALIZING:

711 if not isinstance(v, bool):

712 fail_user_error(f"Bad {k!r} value: {v!r}")

713 finalizing = v

714

715 else:

716 fail_user_error(f"Unknown JSON key: {k!r}")

717

718 if finalizing is None:

719 fail_user_error(f"Missing {TabletParam.FINALIZING!r} JSON key")

720

721 pt_ok, reason = is_candidate_patient_valid_for_group(

722 ptinfo, group, finalizing

723 )

724 if not pt_ok:

725 errors.append(f"{ptinfo} -> {reason}")

726 pt_ok, reason = is_candidate_patient_valid_for_restricted_user(req, ptinfo)

727 if not pt_ok:

728 errors.append(f"{ptinfo} -> {reason}")

729 if errors:

730 fail_user_error(f"Invalid patient: {' // '.join(errors)}")

731

732

733# =============================================================================

734# Extracting information from the POST request

735# =============================================================================

736

737

738 def get_str_var(

739 req: "CamcopsRequest",

740 var: str,

741 mandatory: bool = True,

742 validator: STRING_VALIDATOR_TYPE = validate_anything,

743) -> Optional[str]:

744 """

745 Retrieves a string variable from the CamcopsRequest.

746

747 By default this performs no validation (because, for example, these strings

748 can contain SQL-encoded data or JSON), but there are a number of subsequent

749 operation-specific validation steps.

750

751 Args:

752 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

753 var: name of variable to retrieve

754 mandatory: if ``True``, raise an exception if the variable is missing

755 validator: validator function to use

756

757 Returns:

758 value

759

760 Raises:

761 :exc:`UserErrorException` if the variable was mandatory and

762 no value was provided

763 """

764 try:

765 val = req.get_str_param(var, default=None, validator=validator)

766 if mandatory and val is None:

767 fail_user_error(f"Must provide the variable: {var}")

768 return val

769 except HTTPBadRequest as e: # failed the validator

770 fail_user_error(str(e))

771

772

773 def get_int_var(req: "CamcopsRequest", var: str) -> int:

774 """

775 Retrieves an integer variable from the CamcopsRequest.

776

777 Args:

778 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

779 var: name of variable to retrieve

780

781 Returns:

782 value

783

784 Raises:

785 :exc:`UserErrorException` if no value was provided, or if it wasn't an

786 integer

787 """

788 s = get_str_var(req, var, mandatory=True)

789 try:

790 return int(s)

791 except (TypeError, ValueError):

792 fail_user_error(f"Variable {var} is not a valid integer; was {s!r}")

793

794

795 def get_bool_int_var(req: "CamcopsRequest", var: str) -> bool:

796 """

797 Retrieves a Boolean variable (encoded as an integer) from the

798 CamcopsRequest. Zero represents false; nonzero represents true.

799

800 Args:

801 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

802 var: name of variable to retrieve

803

804 Returns:

805 value

806

807 Raises:

808 :exc:`UserErrorException` if no value was provided, or if it wasn't an

809 integer

810 """

811 num = get_int_var(req, var)

812 return bool(num)

813

814

815 def get_table_from_req(req: "CamcopsRequest", var: str) -> Table:

816 """

817 Retrieves a table name from a HTTP request, checks it's a valid client

818 table, and returns that table.

819

820 Args:

821 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

822 var: variable name (the variable's should be the table name)

823

824 Returns:

825 a SQLAlchemy :class:`Table`

826

827 Raises:

828 :exc:`UserErrorException` if the variable wasn't provided

829 """

830 tablename = get_str_var(req, var, mandatory=True)

831 ensure_valid_table_name(req, tablename)

832 return CLIENT_TABLE_MAP[tablename]

833

834

835 def get_tables_from_post_var(

836 req: "CamcopsRequest", var: str, mandatory: bool = True

837) -> List[Table]:

838 """

839 Gets a list of tables from an HTTP request variable, and ensures all are

840 valid.

841

842 Args:

843 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

844 var: name of variable to retrieve

845 mandatory: if ``True``, raise an exception if the variable is missing

846

847 Returns:

848 a list of SQLAlchemy :class:`Table` objects

849

850 Raises:

851 :exc:`UserErrorException` if the variable was mandatory and

852 no value was provided, or if one or more tables was not valid

853 """

854 cstables = get_str_var(req, var, mandatory=mandatory)

855 if not cstables:

856 return []

857 # can't have any commas in table names, so it's OK to use a simple

858 # split() command

859 tablenames = [x.strip() for x in cstables.split(",")]

860 tables = [] # type: List[Table]

861 for tn in tablenames:

862 ensure_valid_table_name(req, tn)

863 tables.append(CLIENT_TABLE_MAP[tn])

864 return tables

865

866

867 def get_single_field_from_post_var(

868 req: "CamcopsRequest", table: Table, var: str, mandatory: bool = True

869) -> str:

870 """

871 Retrieves a field (column) name from a the request and checks it's not a

872 bad fieldname for the specified table.

873

874 Args:

875 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

876 table: SQLAlchemy :class:`Table` in which the column should exist

877 var: name of variable to retrieve

878 mandatory: if ``True``, raise an exception if the variable is missing

879

880 Returns:

881 the field (column) name

882

883 Raises:

884 :exc:`UserErrorException` if the variable was mandatory and

885 no value was provided, or if the field was not valid for the specified

886 table

887 """

888 field = get_str_var(req, var, mandatory=mandatory)

889 ensure_valid_field_name(table, field)

890 return field

891

892

893 def get_fields_from_post_var(

894 req: "CamcopsRequest",

895 table: Table,

896 var: str,

897 mandatory: bool = True,

898 allowed_nonexistent_fields: List[str] = None,

899) -> List[str]:

900 """

901 Get a comma-separated list of field names from a request and checks that

902 all are acceptable. Returns a list of fieldnames.

903

904 Args:

905 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

906 table: SQLAlchemy :class:`Table` in which the columns should exist

907 var: name of variable to retrieve

908 mandatory: if ``True``, raise an exception if the variable is missing

909 allowed_nonexistent_fields: fields that are allowed to be in the

910 upload but not in the database (special exemptions!)

911

912 Returns:

913 a list of the field (column) names

914

915 Raises:

916 :exc:`UserErrorException` if the variable was mandatory and

917 no value was provided, or if any field was not valid for the specified

918 table

919 """

920 csfields = get_str_var(req, var, mandatory=mandatory)

921 if not csfields:

922 return []

923 allowed_nonexistent_fields = (

924 allowed_nonexistent_fields or []

925 ) # type: List[str]

926 # can't have any commas in fields, so it's OK to use a simple

927 # split() command

928 fields = [x.strip() for x in csfields.split(",")]

929 for f in fields:

930 if f in allowed_nonexistent_fields:

931 continue

932 ensure_valid_field_name(table, f)

933 return fields

934

935

936 def get_values_from_post_var(

937 req: "CamcopsRequest", var: str, mandatory: bool = True

938) -> List[Any]:

939 """

940 Retrieves a list of values from a CSV-separated list of SQL values

941 stored in a CGI form (including e.g. NULL, numbers, quoted strings, and

942 special handling for base-64/hex-encoded BLOBs.)

943

944 Args:

945 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

946 var: name of variable to retrieve

947 mandatory: if ``True``, raise an exception if the variable is missing

948 """

949 csvalues = get_str_var(req, var, mandatory=mandatory)

950 if not csvalues:

951 return []

952 return decode_values(csvalues)

953

954

955 def get_fields_and_values(

956 req: "CamcopsRequest",

957 table: Table,

958 fields_var: str,

959 values_var: str,

960 mandatory: bool = True,

961) -> Dict[str, Any]:

962 """

963 Gets fieldnames and matching values from two variables in a request.

964

965 See :func:`get_fields_from_post_var`, :func:`get_values_from_post_var`.

966

967 Args:

968 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

969 table: SQLAlchemy :class:`Table` in which the columns should exist

970 fields_var: name of CSV "column names" variable to retrieve

971 values_var: name of CSV "corresponding values" variable to retrieve

972 mandatory: if ``True``, raise an exception if the variable is missing

973

974 Returns:

975 a dictionary mapping column names to decoded values

976

977 Raises:

978 :exc:`UserErrorException` if the variable was mandatory and

979 no value was provided, or if any field was not valid for the specified

980 table

981 """

982 fields = get_fields_from_post_var(

983 req, table, fields_var, mandatory=mandatory

984 )

985 values = get_values_from_post_var(req, values_var, mandatory=mandatory)

986 if len(fields) != len(values):

987 fail_user_error(

988 f"Number of fields ({len(fields)}) doesn't match number of values "

989 f"({len(values)})"

990 )

991 return dict(list(zip(fields, values)))

992

993

994 def get_json_from_post_var(

995 req: "CamcopsRequest",

996 key: str,

997 decoder: json.JSONDecoder = None,

998 mandatory: bool = True,

999) -> Any:

1000 """

1001 Returns a Python object from a JSON-encoded value.

1002

1003 Args:

1004 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1005 key: the name of the variable to retrieve

1006 decoder: the JSON decoder object to use; if ``None``, a default is

1007 created

1008 mandatory: if ``True``, raise an exception if the variable is missing

1009

1010 Returns:

1011 Python object, e.g. a list of values, or ``None`` if the object is

1012 invalid and not mandatory

1013

1014 Raises:

1015 :exc:`UserErrorException` if the variable was mandatory and

1016 no value was provided or the value was invalid JSON

1017 """

1018 decoder = decoder or json.JSONDecoder()

1019 j = get_str_var(req, key, mandatory=mandatory) # may raise

1020 if not j: # missing but not mandatory

1021 return None

1022 try:

1023 return decoder.decode(j)

1024 except json.JSONDecodeError:

1025 msg = f"Bad JSON for key {key!r}"

1026 if mandatory:

1027 fail_user_error(msg)

1028 else:

1029 log.warning(msg)

1030 return None

1031

1032

1033# =============================================================================

1034# Sending stuff to the client

1035# =============================================================================

1036

1037

1038 def get_server_id_info(req: "CamcopsRequest") -> Dict[str, str]:

1039 """

1040 Returns a reply for the tablet, as a variable-to-value dictionary, giving

1041 details of the server.

1042 """

1043 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id)

1044 reply = {

1045 TabletParam.DATABASE_TITLE: req.database_title,

1046 TabletParam.ID_POLICY_UPLOAD: group.upload_policy or "",

1047 TabletParam.ID_POLICY_FINALIZE: group.finalize_policy or "",

1048 TabletParam.SERVER_CAMCOPS_VERSION: CAMCOPS_SERVER_VERSION_STRING,

1049 }

1050 for iddef in req.idnum_definitions:

1051 n = iddef.which_idnum

1052 nstr = str(n)

1053 reply[TabletParam.ID_DESCRIPTION_PREFIX + nstr] = (

1054 iddef.description or ""

1055 )

1056 reply[TabletParam.ID_SHORT_DESCRIPTION_PREFIX + nstr] = (

1057 iddef.short_description or ""

1058 )

1059 reply[TabletParam.ID_VALIDATION_METHOD_PREFIX + nstr] = (

1060 iddef.validation_method or ""

1061 )

1062 return reply

1063

1064

1065 def get_select_reply(

1066 fields: Sequence[str], rows: Sequence[Sequence[Any]]

1067) -> Dict[str, str]:

1068 """

1069 Formats the result of a ``SELECT`` query for the client as a dictionary

1070 reply.

1071

1072 Args:

1073 fields: list of field names

1074 rows: list of rows, where each row is a list of values in the same

1075 order as ``fields``

1076

1077 Returns:

1078

1079 a dictionary of the format:

1080

1081 .. code-block:: none

1082

1083 {

1084 "nfields": NUMBER_OF_FIELDS,

1085 "fields": FIELDNAMES_AS_CSV,

1086 "nrecords": NUMBER_OF_RECORDS,

1087 "record0": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES,

1088 ...

1089 "record{nrecords - 1}": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES

1090 }

1091

1092 The final reply to the server is then formatted as text as per

1093 :func:`client_api`.

1094

1095 """ # noqa

1096 nrecords = len(rows)

1097 reply = {

1098 TabletParam.NFIELDS: len(fields),

1099 TabletParam.FIELDS: ",".join(fields),

1100 TabletParam.NRECORDS: nrecords,

1101 }

1102 for r in range(nrecords):

1103 row = rows[r]

1104 encodedvalues = [] # type: List[str]

1105 for val in row:

1106 encodedvalues.append(encode_single_value(val))

1107 reply[TabletParam.RECORD_PREFIX + str(r)] = ",".join(encodedvalues)

1108 return reply # type: ignore[return-value]

1109

1110

1111# =============================================================================

1112# CamCOPS table reading functions

1113# =============================================================================

1114

1115

1116 def record_exists(

1117 req: "CamcopsRequest",

1118 table: Table,

1119 clientpk_name: str,

1120 clientpk_value: Any,

1121) -> ServerRecord:

1122 """

1123 Checks if a record exists, using the device's perspective of a

1124 table/client PK combination.

1125

1126 Args:

1127 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1128 table: an SQLAlchemy :class:`Table`

1129 clientpk_name: the column name of the client's PK

1130 clientpk_value: the client's PK value

1131

1132 Returns:

1133 a :class:`ServerRecord` with the required information

1134

1135 """

1136 query = (

1137 select(

1138 table.c[FN_PK], # server PK

1139 table.c[CLIENT_DATE_FIELD], # when last modified (on the server)

1140 table.c[MOVE_OFF_TABLET_FIELD], # move_off_tablet

1141 )

1142 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)

1143 .where(table.c[FN_CURRENT])

1144 .where(table.c[FN_ERA] == ERA_NOW)

1145 .where(table.c[clientpk_name] == clientpk_value)

1146 )

1147 row = req.dbsession.execute(query).fetchone()

1148 if not row:

1149 return ServerRecord(clientpk_value, False)

1150 server_pk, server_when, move_off_tablet = row

1151 return ServerRecord(

1152 clientpk_value, True, server_pk, server_when, move_off_tablet

1153 )

1154 # Consider a warning/failure if we have >1 row meeting these criteria.

1155 # Not currently checked for.

1156

1157

1158 def client_pks_that_exist(

1159 req: "CamcopsRequest",

1160 table: Table,

1161 clientpk_name: str,

1162 clientpk_values: List[int],

1163) -> Dict[int, ServerRecord]:

1164 """

1165 Searches for client PK values (for this device, current, and 'now')

1166 matching the input list.

1167

1168 Args:

1169 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1170 table: an SQLAlchemy :class:`Table`

1171 clientpk_name: the column name of the client's PK

1172 clientpk_values: a list of the client's PK values

1173

1174 Returns:

1175 a dictionary mapping client_pk to a :class:`ServerRecord` objects, for

1176 those records that match

1177 """

1178 query = (

1179 select(

1180 table.c[FN_PK], # server PK

1181 table.c[clientpk_name], # client PK

1182 table.c[CLIENT_DATE_FIELD], # when last modified (on the server)

1183 table.c[MOVE_OFF_TABLET_FIELD], # move_off_tablet

1184 )

1185 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)

1186 .where(table.c[FN_CURRENT])

1187 .where(table.c[FN_ERA] == ERA_NOW)

1188 .where(table.c[clientpk_name].in_(clientpk_values))

1189 )

1190 rows = req.dbsession.execute(query)

1191 d = {} # type: Dict[int, ServerRecord]

1192 for server_pk, client_pk, server_when, move_off_tablet in rows:

1193 d[client_pk] = ServerRecord(

1194 client_pk, True, server_pk, server_when, move_off_tablet

1195 )

1196 return d

1197

1198

1199 def get_all_predecessor_pks(

1200 req: "CamcopsRequest",

1201 table: Table,

1202 last_pk: int,

1203 include_last: bool = True,

1204) -> List[int]:

1205 """

1206 Retrieves the PKs of all records that are predecessors of the specified one

1207

1208 Args:

1209 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1210 table: an SQLAlchemy :class:`Table`

1211 last_pk: the PK to start with, and work backwards

1212 include_last: include ``last_pk`` in the list

1213

1214 Returns:

1215 the PKs

1216

1217 """

1218 dbsession = req.dbsession

1219 pks = [] # type: List[int]

1220 if include_last:

1221 pks.append(last_pk)

1222 current_pk = last_pk

1223 finished = False

1224 while not finished:

1225 next_pk = dbsession.execute(

1226 select(table.c[FN_PREDECESSOR_PK]).where(

1227 table.c[FN_PK] == current_pk

1228 )

1229 ).scalar() # type: Optional[int]

1230 if next_pk is None:

1231 finished = True

1232 else:

1233 pks.append(next_pk)

1234 current_pk = next_pk

1235 return sorted(pks)

1236

1237

1238# =============================================================================

1239# Record modification functions

1240# =============================================================================

1241

1242

1243 def flag_deleted(

1244 req: "CamcopsRequest",

1245 batchdetails: BatchDetails,

1246 table: Table,

1247 pklist: Iterable[int],

1248) -> None:

1249 """

1250 Marks record(s) as deleted, specified by a list of server PKs within a

1251 table. (Note: "deleted" means "deleted with no successor", not "modified

1252 and replaced by a successor record".)

1253 """

1254 if batchdetails.onestep:

1255 values = values_delete_now(req, batchdetails)

1256 else:

1257 values = values_delete_later()

1258 req.dbsession.execute(

1259 update(table).where(table.c[FN_PK].in_(pklist)).values(values)

1260 )

1261

1262

1263 def flag_all_records_deleted(req: "CamcopsRequest", table: Table) -> int:

1264 """

1265 Marks all records in a table as deleted (that are current and in the

1266 current era).

1267

1268 Returns the number of rows affected.

1269 """

1270 rp = req.dbsession.execute(

1271 update(table)

1272 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)

1273 .where(table.c[FN_CURRENT])

1274 .where(table.c[FN_ERA] == ERA_NOW)

1275 .values(values_delete_later())

1276 ) # type: CursorResult

1277 return rp.rowcount

1278 # https://docs.sqlalchemy.org/en/latest/core/connections.html?highlight=rowcount#sqlalchemy.engine.Result.rowcount # noqa

1279

1280

1281 def flag_deleted_where_clientpk_not(

1282 req: "CamcopsRequest",

1283 table: Table,

1284 clientpk_name: str,

1285 clientpk_values: Sequence[Any],

1286) -> None:

1287 """

1288 Marks for deletion all current/current-era records for a device, within a

1289 specific table, defined by a list of client-side PK values (and the name of

1290 the client-side PK column).

1291 """

1292 rp = req.dbsession.execute(

1293 update(table)

1294 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)

1295 .where(table.c[FN_CURRENT])

1296 .where(table.c[FN_ERA] == ERA_NOW)

1297 .where(table.c[clientpk_name].notin_(clientpk_values))

1298 .values(values_delete_later())

1299 ) # type: CursorResult

1300 if rp.rowcount > 0:

1301 mark_table_dirty(req, table)

1302 # ... but if we are preserving, do NOT mark this table as clean; there may

1303 # be other records that still require preserving.

1304

1305

1306 def flag_modified(

1307 req: "CamcopsRequest",

1308 batchdetails: BatchDetails,

1309 table: Table,

1310 pk: int,

1311 successor_pk: int,

1312) -> None:

1313 """

1314 Marks a record as old, storing its successor's details.

1315

1316 Args:

1317 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1318 batchdetails: the :class:`BatchDetails`

1319 table: SQLAlchemy :class:`Table`

1320 pk: server PK of the record to mark as old

1321 successor_pk: server PK of its successor

1322 """

1323 if batchdetails.onestep:

1324 req.dbsession.execute(

1325 update(table)

1326 .where(table.c[FN_PK] == pk)

1327 .values(

1328 {

1329 FN_CURRENT: 0,

1330 FN_REMOVAL_PENDING: 0,

1331 FN_SUCCESSOR_PK: successor_pk,

1332 FN_REMOVING_USER_ID: req.user_id,

1333 FN_WHEN_REMOVED_EXACT: req.now,

1334 FN_WHEN_REMOVED_BATCH_UTC: batchdetails.batchtime,

1335 }

1336 )

1337 )

1338 else:

1339 req.dbsession.execute(

1340 update(table)

1341 .where(table.c[FN_PK] == pk)

1342 .values({FN_REMOVAL_PENDING: 1, FN_SUCCESSOR_PK: successor_pk})

1343 )

1344

1345

1346 def flag_multiple_records_for_preservation(

1347 req: "CamcopsRequest",

1348 batchdetails: BatchDetails,

1349 table: Table,

1350 pks_to_preserve: List[int],

1351) -> None:

1352 """

1353 Low-level function to mark records for preservation by server PK.

1354 Does not concern itself with the predecessor chain (for which, see

1355 :func:`flag_record_for_preservation`).

1356

1357 Args:

1358 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1359 batchdetails: the :class:`BatchDetails`

1360 table: SQLAlchemy :class:`Table`

1361 pks_to_preserve: server PK of the records to mark as preserved

1362 """

1363 if batchdetails.onestep:

1364 req.dbsession.execute(

1365 update(table)

1366 .where(table.c[FN_PK].in_(pks_to_preserve))

1367 .values(values_preserve_now(req, batchdetails))

1368 )

1369 # Also any associated special notes:

1370 new_era = batchdetails.new_era

1371 # noinspection PyUnresolvedReferences

1372 req.dbsession.execute(

1373 update(SpecialNote.__table__) # type: ignore[arg-type]

1374 .where(SpecialNote.basetable == table.name)

1375 .where(SpecialNote.device_id == req.tabletsession.device_id)

1376 .where(SpecialNote.era == ERA_NOW)

1377 .where(

1378 exists()

1379 .select_from(table)

1380 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id)

1381 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id)

1382 .where(table.c[FN_ERA] == new_era)

1383 )

1384 # ^^^^^^^^^^^^^^^^^^^^^^^^^^

1385 # This bit restricts to records being preserved.

1386 .values(era=new_era)

1387 )

1388 else:

1389 req.dbsession.execute(

1390 update(table)

1391 .where(table.c[FN_PK].in_(pks_to_preserve))

1392 .values({MOVE_OFF_TABLET_FIELD: 1})

1393 )

1394

1395

1396 def flag_record_for_preservation(

1397 req: "CamcopsRequest", batchdetails: BatchDetails, table: Table, pk: int

1398) -> List[int]:

1399 """

1400 Marks a record for preservation (moving off the tablet, changing its

1401 era details).

1402

1403 2018-11-18: works back through the predecessor chain too, fixing an old

1404 bug.

1405

1406 Args:

1407 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1408 batchdetails: the :class:`BatchDetails`

1409 table: SQLAlchemy :class:`Table`

1410 pk: server PK of the record to mark

1411

1412 Returns:

1413 list: all PKs being preserved

1414 """

1415 pks_to_preserve = get_all_predecessor_pks(req, table, pk)

1416 flag_multiple_records_for_preservation(

1417 req, batchdetails, table, pks_to_preserve

1418 )

1419 return pks_to_preserve

1420

1421

1422 def preserve_all(

1423 req: "CamcopsRequest", batchdetails: BatchDetails, table: Table

1424) -> None:

1425 """

1426 Preserves all records in a table for a device, including non-current ones.

1427

1428 Args:

1429 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1430 batchdetails: the :class:`BatchDetails`

1431 table: SQLAlchemy :class:`Table`

1432 """

1433 device_id = req.tabletsession.device_id

1434 req.dbsession.execute(

1435 update(table)

1436 .where(table.c[FN_DEVICE_ID] == device_id)

1437 .where(table.c[FN_ERA] == ERA_NOW)

1438 .values(values_preserve_now(req, batchdetails))

1439 )

1440

1441

1442# =============================================================================

1443# Upload helper functions

1444# =============================================================================

1445

1446

1447 def process_upload_record_special(

1448 req: "CamcopsRequest",

1449 batchdetails: BatchDetails,

1450 table: Table,

1451 valuedict: Dict[str, Any],

1452) -> None:

1453 """

1454 Special processing function for upload, in which we inspect the data.

1455 Called by :func:`upload_record_core`.

1456

1457 1. Handles old clients with ID information in the patient table, etc.

1458 (Note: this can be IGNORED for any client using

1459 :func:`op_upload_entire_database`, as these are newer.)

1460

1461 2. Validates ID numbers.

1462

1463 Args:

1464 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1465 batchdetails: the :class:`BatchDetails`

1466 table: an SQLAlchemy :class:`Table`

1467 valuedict: a dictionary of {colname: value} pairs from the client.

1468 May be modified.

1469 """

1470 ts = req.tabletsession

1471 tablename = table.name

1472

1473 if tablename == Patient.__tablename__:

1474 # ---------------------------------------------------------------------

1475 # Deal with old tablets that had ID numbers in a less flexible format.

1476 # ---------------------------------------------------------------------

1477 if ts.cope_with_deleted_patient_descriptors:

1478 # Old tablets (pre-2.0.0) will upload copies of the ID

1479 # descriptions with the patient. To cope with that, we

1480 # remove those here:

1481 for n in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):

1482 nstr = str(n)

1483 fn_desc = FP_ID_DESC + nstr

1484 fn_shortdesc = FP_ID_SHORT_DESC + nstr

1485 valuedict.pop(fn_desc, None) # remove item, if exists

1486 valuedict.pop(fn_shortdesc, None)

1487

1488 if ts.cope_with_old_idnums:

1489 # Insert records into the new ID number table from the old

1490 # patient table:

1491 for which_idnum in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):

1492 nstr = str(which_idnum)

1493 fn_idnum = FP_ID_NUM + nstr

1494 idnum_value = valuedict.pop(fn_idnum, None)

1495 # ... and remove it from our new Patient record

1496 patient_id = valuedict.get("id", None)

1497 if idnum_value is None or patient_id is None:

1498 continue

1499 # noinspection PyUnresolvedReferences

1500 mark_table_dirty(req, PatientIdNum.__table__) # type: ignore[arg-type] # noqa: E501

1501 client_date_value = coerce_to_pendulum(

1502 valuedict[CLIENT_DATE_FIELD]

1503 )

1504 # noinspection PyUnresolvedReferences

1505 upload_record_core(

1506 req=req,

1507 batchdetails=batchdetails,

1508 table=PatientIdNum.__table__, # type: ignore[arg-type]

1509 clientpk_name="id",

1510 valuedict={

1511 "id": fake_tablet_id_for_patientidnum(

1512 patient_id=patient_id, which_idnum=which_idnum

1513 ), # ... guarantees a pseudo client PK

1514 "patient_id": patient_id,

1515 "which_idnum": which_idnum,

1516 "idnum_value": idnum_value,

1517 CLIENT_DATE_FIELD: client_date_value,

1518 MOVE_OFF_TABLET_FIELD: valuedict[

1519 MOVE_OFF_TABLET_FIELD

1520 ],

1521 },

1522 )

1523 # Now, how to deal with deletion, i.e. records missing from the

1524 # tablet? See our caller, op_upload_table(), which has a special

1525 # handler for this.

1526 #

1527 # Note that op_upload_record() is/was only used for BLOBs, so we

1528 # don't have to worry about special processing for that aspect

1529 # here; also, that method handles deletion in a different way.

1530

1531 elif tablename == PatientIdNum.__tablename__:

1532 # ---------------------------------------------------------------------

1533 # Validate ID numbers.

1534 # ---------------------------------------------------------------------

1535 which_idnum = valuedict.get("which_idnum", None)

1536 if which_idnum not in req.valid_which_idnums:

1537 fail_user_error(f"No such ID number type: {which_idnum}")

1538 idnum_value = valuedict.get("idnum_value", None)

1539 if not req.is_idnum_valid(which_idnum, idnum_value):

1540 why_invalid = req.why_idnum_invalid(which_idnum, idnum_value)

1541 fail_user_error(

1542 f"For ID type {which_idnum}, ID number {idnum_value} is "

1543 f"invalid: {why_invalid}"

1544 )

1545

1546

1547 def upload_record_core(

1548 req: "CamcopsRequest",

1549 batchdetails: BatchDetails,

1550 table: Table,

1551 clientpk_name: str,

1552 valuedict: Dict[str, Any],

1553 server_live_current_records: List[ServerRecord] = None,

1554) -> UploadRecordResult:

1555 """

1556 Uploads a record. Deals with IDENTICAL, NEW, and MODIFIED records.

1557

1558 Used by :func:`upload_table` and :func:`upload_record`.

1559

1560 Args:

1561 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1562 batchdetails: the :class:`BatchDetails`

1563 table: an SQLAlchemy :class:`Table`

1564 clientpk_name: the column name of the client's PK

1565 valuedict: a dictionary of {colname: value} pairs from the client

1566 server_live_current_records: list of :class:`ServerRecord` objects for

1567 the active records on the server for this client, in this table

1568

1569 Returns:

1570 a :class:`UploadRecordResult` object

1571 """

1572 require_keys(

1573 valuedict, [clientpk_name, CLIENT_DATE_FIELD, MOVE_OFF_TABLET_FIELD]

1574 )

1575 clientpk_value = valuedict[clientpk_name]

1576

1577 if server_live_current_records:

1578 # All server records for this table/device/era have been prefetched.

1579 serverrec = next(

1580 (

1581 r

1582 for r in server_live_current_records

1583 if r.client_pk == clientpk_value

1584 ),

1585 None,

1586 )

1587 if serverrec is None:

1588 serverrec = ServerRecord(clientpk_value, False)

1589 else:

1590 # Look up this record specifically.

1591 serverrec = record_exists(req, table, clientpk_name, clientpk_value)

1592

1593 if DEBUG_UPLOAD:

1594 log.debug("upload_record_core: {}, {}", table.name, serverrec)

1595

1596 oldserverpk = serverrec.server_pk

1597 urr = UploadRecordResult(

1598 oldserverpk=oldserverpk,

1599 specifically_marked_for_preservation=bool(

1600 valuedict[MOVE_OFF_TABLET_FIELD]

1601 ),

1602 dirty=True,

1603 )

1604 if serverrec.exists:

1605 # There's an existing record, which is either identical or not.

1606 client_date_value = coerce_to_pendulum(valuedict[CLIENT_DATE_FIELD])

1607 if serverrec.server_when == client_date_value:

1608 # The existing record is identical.

1609 # No action needed unless MOVE_OFF_TABLET_FIELDNAME is set.

1610 if not urr.specifically_marked_for_preservation:

1611 urr.dirty = False

1612 else:

1613 # The existing record is different. We need a logical UPDATE, but

1614 # maintaining an audit trail.

1615 process_upload_record_special(req, batchdetails, table, valuedict)

1616 urr.newserverpk = insert_record(

1617 req, batchdetails, table, valuedict, oldserverpk

1618 )

1619 flag_modified(

1620 req, batchdetails, table, oldserverpk, urr.newserverpk

1621 )

1622 else:

1623 # The record is NEW. We need to INSERT it.

1624 process_upload_record_special(req, batchdetails, table, valuedict)

1625 urr.newserverpk = insert_record(

1626 req, batchdetails, table, valuedict, None

1627 )

1628 if urr.specifically_marked_for_preservation:

1629 preservation_pks = flag_record_for_preservation(

1630 req, batchdetails, table, urr.latest_pk

1631 )

1632 urr.note_specifically_marked_preservation_pks(preservation_pks)

1633

1634 if DEBUG_UPLOAD:

1635 log.debug("upload_record_core: {}, {!r}", table.name, urr)

1636 return urr

1637

1638

1639 def insert_record(

1640 req: "CamcopsRequest",

1641 batchdetails: BatchDetails,

1642 table: Table,

1643 valuedict: Dict[str, Any],

1644 predecessor_pk: Optional[int],

1645) -> int:

1646 """

1647 Inserts a record, or raises an exception if that fails.

1648

1649 Args:

1650 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1651 batchdetails: the :class:`BatchDetails`

1652 table: an SQLAlchemy :class:`Table`

1653 valuedict: a dictionary of {colname: value} pairs from the client

1654 predecessor_pk: an optional server PK of the record's predecessor

1655

1656 Returns:

1657 the server PK of the new record

1658 """

1659 ts = req.tabletsession

1660 valuedict.update(

1661 {

1662 FN_DEVICE_ID: ts.device_id,

1663 FN_ERA: ERA_NOW,

1664 FN_REMOVAL_PENDING: 0,

1665 FN_PREDECESSOR_PK: predecessor_pk,

1666 FN_CAMCOPS_VERSION: ts.tablet_version_str,

1667 FN_GROUP_ID: req.user.upload_group_id,

1668 }

1669 )

1670 if batchdetails.onestep:

1671 valuedict.update(

1672 {

1673 FN_CURRENT: 1,

1674 FN_ADDITION_PENDING: 0,

1675 FN_ADDING_USER_ID: req.user_id,

1676 FN_WHEN_ADDED_EXACT: req.now,

1677 FN_WHEN_ADDED_BATCH_UTC: batchdetails.batchtime,

1678 }

1679 )

1680 else:

1681 valuedict.update({FN_CURRENT: 0, FN_ADDITION_PENDING: 1})

1682 rp = req.dbsession.execute(

1683 table.insert().values(valuedict)

1684 ) # type: CursorResult

1685 # In SQLAlchemy 1.3, execute() returned a ResultProxy, and after an

1686 # insert() call, ResultProxy.inserted_primary_key was a list of scalars,

1687 # corresponding to the list of primary key columns in the target table,

1688 # representing the primary key of the row just inserted (a list because a

1689 # primary key can be a composite of many columns) [1]. We then asserted it

1690 # was a list of length 1, and returned the first element. In SQLAlchemy

1691 # 1.4+, we get a CursorResult back instead, and its inserted_primary_key is

1692 # a named tuple of primary key values, for that single inserted row [2] (or

1693 # None if there was not a valid single-row insert, or raises an exception

1694 # after a multi-row insert) [3]. The previous length check was likely an

1695 # inaccurate attempt to check that 1 row had been inserted (rather than

1696 # that there was 1 primary key column). If the insert fails, however, the

1697 # database call will raise an exception; and even if it didn't, the attempt

1698 # to access rp.inserted_primary_key[0] as None[0] would raise a TypeError.

1699 # So the additional assertion was a waste of time.

1700 # [1] https://docs.sqlalchemy.org/en/13/core/connections.html#sqlalchemy.engine.ResultProxy.inserted_primary_key # noqa: E501

1701 # [2] https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.BaseCursorResult.inserted_primary_key # noqa: E501

1702 # [3] see sqlalchemy/engine/cursor.py

1703 return rp.inserted_primary_key[0]

1704

1705

1706 def audit_upload(

1707 req: "CamcopsRequest", changes: List[UploadTableChanges]

1708) -> None:

1709 """

1710 Writes audit information for an upload.

1711

1712 Args:

1713 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1714 changes: a list of :class:`UploadTableChanges` objects, one per table

1715 """

1716 msg = (

1717 f"Upload from device {req.tabletsession.device_id}, "

1718 f"username {req.tabletsession.username!r}: "

1719 )

1720 changes = [x for x in changes if x.any_changes]

1721 if changes:

1722 changes.sort(key=lambda x: x.tablename)

1723 msg += ", ".join(x.description() for x in changes)

1724 else:

1725 msg += "No changes"

1726 log.info("audit_upload: {}", msg)

1727 audit(req, msg)

1728

1729

1730# =============================================================================

1731# Batch (atomic) upload and preserving

1732# =============================================================================

1733

1734

1735 def get_batch_details(req: "CamcopsRequest") -> BatchDetails:

1736 """

1737 Returns the :class:`BatchDetails` for the current upload. If none exists,

1738 a new batch is created and returned.

1739

1740 SIDE EFFECT: if the username is different from the username that started

1741 a previous upload batch for this device, we restart the upload batch (thus

1742 rolling back previous pending changes).

1743

1744 Raises:

1745 :exc:`camcops_server.cc_modules.cc_client_api_core.ServerErrorException`

1746 if the device doesn't exist

1747 """

1748 device_id = req.tabletsession.device_id

1749 # noinspection PyUnresolvedReferences

1750 query = (

1751 select(

1752 Device.ongoing_upload_batch_utc,

1753 Device.uploading_user_id,

1754 Device.currently_preserving,

1755 )

1756 .select_from(Device.__table__)

1757 .where(Device.id == device_id)

1758 )

1759 row = req.dbsession.execute(query).fetchone()

1760 if not row:

1761 fail_server_error(

1762 f"Device {device_id} missing from Device table"

1763 ) # will raise

1764 upload_batch_utc, uploading_user_id, currently_preserving = row

1765 if not upload_batch_utc or uploading_user_id != req.user_id:

1766 # SIDE EFFECT: if the username changes, we restart (and thus roll back

1767 # previous pending changes)

1768 start_device_upload_batch(req)

1769 return BatchDetails(req.now_utc, False)

1770 return BatchDetails(upload_batch_utc, currently_preserving)

1771

1772

1773 def start_device_upload_batch(req: "CamcopsRequest") -> None:

1774 """

1775 Starts an upload batch for a device.

1776 """

1777 rollback_all(req)

1778 # noinspection PyUnresolvedReferences

1779 req.dbsession.execute(

1780 update(Device.__table__) # type: ignore[arg-type]

1781 .where(Device.id == req.tabletsession.device_id)

1782 .values(

1783 last_upload_batch_utc=req.now_utc,

1784 ongoing_upload_batch_utc=req.now_utc,

1785 uploading_user_id=req.tabletsession.user_id,

1786 )

1787 )

1788

1789

1790 def _clear_ongoing_upload_batch_details(req: "CamcopsRequest") -> None:

1791 """

1792 Clears upload batch details from the Device table.

1793

1794 Args:

1795 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1796 """

1797 # noinspection PyUnresolvedReferences

1798 req.dbsession.execute(

1799 update(Device.__table__) # type: ignore[arg-type]

1800 .where(Device.id == req.tabletsession.device_id)

1801 .values(

1802 ongoing_upload_batch_utc=None,

1803 uploading_user_id=None,

1804 currently_preserving=0,

1805 )

1806 )

1807

1808

1809 def end_device_upload_batch(

1810 req: "CamcopsRequest", batchdetails: BatchDetails

1811) -> None:

1812 """

1813 Ends an upload batch, committing all changes made thus far.

1814

1815 Args:

1816 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1817 batchdetails: the :class:`BatchDetails`

1818 """

1819 commit_all(req, batchdetails)

1820 _clear_ongoing_upload_batch_details(req)

1821

1822

1823 def clear_device_upload_batch(req: "CamcopsRequest") -> None:

1824 """

1825 Ensures there is nothing pending. Rools back previous changes. Wipes any

1826 ongoing batch details.

1827

1828 Args:

1829 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1830 """

1831 rollback_all(req)

1832 _clear_ongoing_upload_batch_details(req)

1833

1834

1835 def start_preserving(req: "CamcopsRequest") -> None:

1836 """

1837 Starts preservation (the process of moving records from the NOW era to

1838 an older era, so they can be removed safely from the tablet).

1839

1840 Called by :func:`op_start_preservation`.

1841

1842 In this situation, we start by assuming that ALL tables are "dirty",

1843 because they may have live records from a previous upload.

1844 """

1845 # noinspection PyUnresolvedReferences

1846 req.dbsession.execute(

1847 update(Device.__table__) # type: ignore[arg-type]

1848 .where(Device.id == req.tabletsession.device_id)

1849 .values(currently_preserving=1)

1850 )

1851 mark_all_tables_dirty(req)

1852

1853

1854 def mark_table_dirty(req: "CamcopsRequest", table: Table) -> None:

1855 """

1856 Marks a table as having been modified during the current upload.

1857 """

1858 tablename = table.name

1859 device_id = req.tabletsession.device_id

1860 dbsession = req.dbsession

1861 # noinspection PyUnresolvedReferences

1862 table_already_dirty = exists_in_table(

1863 dbsession,

1864 DirtyTable.__table__, # type: ignore[arg-type]

1865 DirtyTable.device_id == device_id,

1866 DirtyTable.tablename == tablename,

1867 )

1868 if not table_already_dirty:

1869 # noinspection PyUnresolvedReferences

1870 dbsession.execute(

1871 DirtyTable.__table__.insert().values( # type: ignore[attr-defined]

1872 device_id=device_id, tablename=tablename

1873 )

1874 )

1875

1876

1877 def mark_tables_dirty(req: "CamcopsRequest", tables: List[Table]) -> None:

1878 """

1879 Marks multiple tables as dirty.

1880 """

1881 if not tables:

1882 return

1883 device_id = req.tabletsession.device_id

1884 tablenames = [t.name for t in tables]

1885 # Delete first

1886 # noinspection PyUnresolvedReferences

1887 req.dbsession.execute(

1888 DirtyTable.__table__.delete() # type: ignore[attr-defined]

1889 .where(DirtyTable.device_id == device_id)

1890 .where(DirtyTable.tablename.in_(tablenames))

1891 )

1892 # Then insert

1893 insert_values = [

1894 {"device_id": device_id, "tablename": tn} for tn in tablenames

1895 ]

1896 # noinspection PyUnresolvedReferences

1897 req.dbsession.execute(DirtyTable.__table__.insert(), insert_values) # type: ignore[attr-defined] # noqa: E501

1898

1899

1900 def mark_all_tables_dirty(req: "CamcopsRequest") -> None:

1901 """

1902 If we are preserving, we assume that all tables are "dirty" (require work

1903 when we complete the upload) unless we specifically mark them clean.

1904 """

1905 device_id = req.tabletsession.device_id

1906 # Delete first

1907 # noinspection PyUnresolvedReferences

1908 req.dbsession.execute(

1909 DirtyTable.__table__.delete().where(DirtyTable.device_id == device_id) # type: ignore[attr-defined] # noqa: E501

1910 )

1911 # Now insert

1912 # https://docs.sqlalchemy.org/en/latest/core/tutorial.html#execute-multiple

1913 all_client_tablenames = list(CLIENT_TABLE_MAP.keys())

1914 insert_values = [

1915 {"device_id": device_id, "tablename": tn}

1916 for tn in all_client_tablenames

1917 ]

1918 # noinspection PyUnresolvedReferences

1919 req.dbsession.execute(DirtyTable.__table__.insert(), insert_values) # type: ignore[attr-defined] # noqa: E501

1920

1921

1922 def mark_table_clean(req: "CamcopsRequest", table: Table) -> None:

1923 """

1924 Marks a table as being clean: that is,

1925

1926 - the table has been scanned during the current upload

1927 - there is nothing to do (either from the current upload, OR A PREVIOUS

1928 UPLOAD).

1929 """

1930 tablename = table.name

1931 device_id = req.tabletsession.device_id

1932 # noinspection PyUnresolvedReferences

1933 req.dbsession.execute(

1934 DirtyTable.__table__.delete() # type: ignore[attr-defined]

1935 .where(DirtyTable.device_id == device_id)

1936 .where(DirtyTable.tablename == tablename)

1937 )

1938

1939

1940 def mark_tables_clean(req: "CamcopsRequest", tables: List[Table]) -> None:

1941 """

1942 Marks multiple tables as clean.

1943 """

1944 if not tables:

1945 return

1946 device_id = req.tabletsession.device_id

1947 tablenames = [t.name for t in tables]

1948 # Delete first

1949 # noinspection PyUnresolvedReferences

1950 req.dbsession.execute(

1951 DirtyTable.__table__.delete() # type: ignore[attr-defined]

1952 .where(DirtyTable.device_id == device_id)

1953 .where(DirtyTable.tablename.in_(tablenames))

1954 )

1955

1956

1957 def get_dirty_tables(req: "CamcopsRequest") -> List[Table]:

1958 """

1959 Returns tables marked as dirty for this device. (See

1960 :func:`mark_table_dirty`.)

1961 """

1962 query = select(DirtyTable.tablename).where(

1963 DirtyTable.device_id == req.tabletsession.device_id

1964 )

1965 tablenames = fetch_all_first_values(req.dbsession, query)

1966 return [CLIENT_TABLE_MAP[tn] for tn in tablenames]

1967

1968

1969 def commit_all(req: "CamcopsRequest", batchdetails: BatchDetails) -> None:

1970 """

1971 Commits additions, removals, and preservations for all tables.

1972

1973 Args:

1974 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

1975 batchdetails: the :class:`BatchDetails`

1976 """

1977 tables = get_dirty_tables(req)

1978 # log.debug("Dirty tables: {}", list(t.name for t in tables))

1979 tables.sort(key=upload_commit_order_sorter)

1980

1981 changelist = [] # type: List[UploadTableChanges]

1982 for table in tables:

1983 auditinfo = commit_table(req, batchdetails, table, clear_dirty=False)

1984 changelist.append(auditinfo)

1985

1986 if batchdetails.preserving:

1987 # Also preserve/finalize any corresponding special notes (2015-02-01),

1988 # but all in one go (2018-11-13).

1989 # noinspection PyUnresolvedReferences

1990 req.dbsession.execute(

1991 update(SpecialNote.__table__) # type: ignore[arg-type]

1992 .where(SpecialNote.device_id == req.tabletsession.device_id)

1993 .where(SpecialNote.era == ERA_NOW)

1994 .values(era=batchdetails.new_era)

1995 )

1996

1997 clear_dirty_tables(req)

1998 audit_upload(req, changelist)

1999

2000 # Performance 2018-11-13:

2001 # - start at 2.407 s

2002 # - remove final temptable clearance and COUNT(*): 1.626 to 2.118 s

2003 # - IN clause using Python literal not temptable: 1.18 to 1.905 s

2004 # - minor tidy: 1.075 to 1.65

2005 # - remove ORDER BY from task indexing: 1.093 to 1.607

2006 # - optimize special note code won't affect this: 1.076 to 1.617

2007 # At this point, entire upload process ~5s.

2008 # - big difference from commit_table() query optimization

2009 # - huge difference from being more careful with mark_table_dirty()

2010 # - further table scanning optimizations: fewer queries

2011 # Overall upload down to ~2.4s

2012

2013

2014 def commit_table(

2015 req: "CamcopsRequest",

2016 batchdetails: BatchDetails,

2017 table: Table,

2018 clear_dirty: bool = True,

2019) -> UploadTableChanges:

2020 """

2021 Commits additions, removals, and preservations for one table.

2022

2023 Should ONLY be called by :func:`commit_all`.

2024

2025 Also updates task indexes.

2026

2027 Args:

2028 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

2029 batchdetails: the :class:`BatchDetails`

2030 table: SQLAlchemy :class:`Table`

2031 clear_dirty: remove the table from the record of dirty tables for

2032 this device? (If called from :func:`commit_all`, this should be

2033 ``False``, since it's faster to clear all dirty tables for the

2034 device simultaneously than one-by-one.)

2035

2036 Returns:

2037 an :class:`UploadTableChanges` object

2038 """

2039

2040 # Tried storing PKs in temporary tables, rather than using an IN clause

2041 # with Python values, as per

2042 # https://www.xaprb.com/blog/2006/06/28/why-large-in-clauses-are-problematic/ # noqa

2043 # However, it was slow.

2044 # We can gain a lot of efficiency (empirically) by:

2045 # - Storing PKs in Python

2046 # - Only performing updates when we need to

2047 # - Using a single query per table to get "add/remove/preserve" PKs

2048

2049 # -------------------------------------------------------------------------

2050 # Helpful temporary variables

2051 # -------------------------------------------------------------------------

2052 user_id = req.user_id

2053 device_id = req.tabletsession.device_id

2054 exacttime = req.now

2055 dbsession = req.dbsession

2056 tablename = table.name

2057 batchtime = batchdetails.batchtime

2058 preserving = batchdetails.preserving

2059

2060 # -------------------------------------------------------------------------

2061 # Fetch addition, removal, preservation, current PKs in a single query

2062 # -------------------------------------------------------------------------

2063 tablechanges = UploadTableChanges(table)

2064 serverrecs = get_server_live_records(

2065 req, device_id, table, current_only=False

2066 )

2067 for sr in serverrecs:

2068 tablechanges.note_serverrec(sr, preserving=preserving)

2069

2070 # -------------------------------------------------------------------------

2071 # Additions

2072 # -------------------------------------------------------------------------

2073 # Update the records we're adding

2074 addition_pks = tablechanges.addition_pks

2075 if addition_pks:

2076 # log.debug("commit_table: {}, adding server PKs {}",

2077 # tablename, addition_pks)

2078 dbsession.execute(

2079 update(table)

2080 .where(table.c[FN_PK].in_(addition_pks))

2081 .values(

2082 {

2083 FN_CURRENT: 1,

2084 FN_ADDITION_PENDING: 0,

2085 FN_ADDING_USER_ID: user_id,

2086 FN_WHEN_ADDED_EXACT: exacttime,

2087 FN_WHEN_ADDED_BATCH_UTC: batchtime,

2088 }

2089 )

2090 )

2091

2092 # -------------------------------------------------------------------------

2093 # Removals

2094 # -------------------------------------------------------------------------

2095 # Update the records we're removing

2096 removal_pks = tablechanges.removal_pks

2097 if removal_pks:

2098 # log.debug("commit_table: {}, removing server PKs {}",

2099 # tablename, removal_pks)

2100 dbsession.execute(

2101 update(table)

2102 .where(table.c[FN_PK].in_(removal_pks))

2103 .values(values_delete_now(req, batchdetails))

2104 )

2105

2106 # -------------------------------------------------------------------------

2107 # Preservation

2108 # -------------------------------------------------------------------------

2109 # Preserve necessary records

2110 preservation_pks = tablechanges.preservation_pks

2111 if preservation_pks:

2112 # log.debug("commit_table: {}, preserving server PKs {}",

2113 # tablename, preservation_pks)

2114 new_era = batchdetails.new_era

2115 dbsession.execute(

2116 update(table)

2117 .where(table.c[FN_PK].in_(preservation_pks))

2118 .values(values_preserve_now(req, batchdetails))

2119 )

2120 if not preserving:

2121 # Also preserve/finalize any corresponding special notes

2122 # (2015-02-01), just for records being specifically preserved. If

2123 # we are preserving, this step happens in one go in commit_all()

2124 # (2018-11-13).

2125 # noinspection PyUnresolvedReferences

2126 dbsession.execute(

2127 update(SpecialNote.__table__) # type: ignore[arg-type]

2128 .where(SpecialNote.basetable == tablename)

2129 .where(SpecialNote.device_id == device_id)

2130 .where(SpecialNote.era == ERA_NOW)

2131 .where(

2132 exists()

2133 .select_from(table)

2134 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id)

2135 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id)

2136 .where(table.c[FN_ERA] == new_era)

2137 )

2138 # ^^^^^^^^^^^^^^^^^^^^^^^^^^

2139 # This bit restricts to records being preserved.

2140 .values(era=new_era)

2141 )

2142

2143 # -------------------------------------------------------------------------

2144 # Update special indexes

2145 # -------------------------------------------------------------------------

2146 update_indexes_and_push_exports(req, batchdetails, tablechanges)

2147

2148 # -------------------------------------------------------------------------

2149 # Remove individually from list of dirty tables?

2150 # -------------------------------------------------------------------------

2151 if clear_dirty:

2152 # noinspection PyUnresolvedReferences

2153 dbsession.execute(

2154 DirtyTable.__table__.delete() # type: ignore[attr-defined]

2155 .where(DirtyTable.device_id == device_id)

2156 .where(DirtyTable.tablename == tablename)

2157 )

2158 # ... otherwise a call to clear_dirty_tables() must be made.

2159

2160 if DEBUG_UPLOAD:

2161 log.debug("commit_table: {}", tablechanges)

2162

2163 return tablechanges

2164

2165

2166 def rollback_all(req: "CamcopsRequest") -> None:

2167 """

2168 Rolls back all pending changes for a device.

2169 """

2170 tables = get_dirty_tables(req)

2171 for table in tables:

2172 rollback_table(req, table)

2173 clear_dirty_tables(req)

2174

2175

2176 def rollback_table(req: "CamcopsRequest", table: Table) -> None:

2177 """

2178 Rolls back changes for an individual table for a device.

2179 """

2180 device_id = req.tabletsession.device_id

2181 # Pending additions

2182 req.dbsession.execute(

2183 table.delete()

2184 .where(table.c[FN_DEVICE_ID] == device_id)

2185 .where(table.c[FN_ADDITION_PENDING])

2186 )

2187 # Pending deletions

2188 req.dbsession.execute(

2189 update(table)

2190 .where(table.c[FN_DEVICE_ID] == device_id)

2191 .where(table.c[FN_REMOVAL_PENDING])

2192 .values(

2193 {

2194 FN_REMOVAL_PENDING: 0,

2195 FN_WHEN_ADDED_EXACT: None,

2196 FN_WHEN_REMOVED_BATCH_UTC: None,

2197 FN_REMOVING_USER_ID: None,

2198 FN_SUCCESSOR_PK: None,

2199 }

2200 )

2201 )

2202 # Record-specific preservation (set by flag_record_for_preservation())

2203 req.dbsession.execute(

2204 update(table)

2205 .where(table.c[FN_DEVICE_ID] == device_id)

2206 .values({MOVE_OFF_TABLET_FIELD: 0})

2207 )

2208

2209

2210 def clear_dirty_tables(req: "CamcopsRequest") -> None:

2211 """

2212 Clears the dirty-table list for a device.

2213 """

2214 device_id = req.tabletsession.device_id

2215 # noinspection PyUnresolvedReferences

2216 req.dbsession.execute(

2217 DirtyTable.__table__.delete().where(DirtyTable.device_id == device_id) # type: ignore[attr-defined] # noqa: E501

2218 )

2219

2220

2221# =============================================================================

2222# Additional helper functions for one-step upload

2223# =============================================================================

2224

2225

2226 def process_table_for_onestep_upload(

2227 req: "CamcopsRequest",

2228 batchdetails: BatchDetails,

2229 table: Table,

2230 clientpk_name: str,

2231 rows: List[Dict[str, Any]],

2232) -> UploadTableChanges:

2233 """

2234 Performs all upload steps for a table.

2235

2236 Note that we arrive here in a specific and safe table order; search for

2237 :func:`camcops_server.cc_modules.cc_client_api_helpers.upload_commit_order_sorter`.

2238

2239 Args:

2240 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

2241 batchdetails: the :class:`BatchDetails`

2242 table: an SQLAlchemy :class:`Table`

2243 clientpk_name: the name of the PK field on the client

2244 rows: a list of rows, where each row is a dictionary mapping field

2245 (column) names to values (those values being encoded as SQL-style

2246 literals in our extended syntax)

2247

2248 Returns:

2249 an :class:`UploadTableChanges` object

2250 """

2251 serverrecs = get_server_live_records(

2252 req,

2253 req.tabletsession.device_id,

2254 table,

2255 clientpk_name,

2256 current_only=False,

2257 )

2258 servercurrentrecs = [r for r in serverrecs if r.current]

2259 if rows and not clientpk_name:

2260 fail_user_error(

2261 f"Client-side PK name not specified by client for "

2262 f"non-empty table {table.name!r}"

2263 )

2264 tablechanges = UploadTableChanges(table)

2265 server_pks_uploaded = [] # type: List[int]

2266 for row in rows:

2267 valuedict = {k: decode_single_value(v) for k, v in row.items()}

2268 urr = upload_record_core(

2269 req,

2270 batchdetails,

2271 table,

2272 clientpk_name,

2273 valuedict,

2274 server_live_current_records=servercurrentrecs,

2275 )

2276 # ... handles addition, modification, preservation, special processing

2277 # But we also make a note of these for indexing:

2278 if urr.oldserverpk is not None:

2279 server_pks_uploaded.append(urr.oldserverpk)

2280 tablechanges.note_urr(

2281 urr, preserving_new_records=batchdetails.preserving

2282 )

2283 # Which leaves:

2284 # (*) Deletion (where no record was uploaded at all)

2285 server_pks_for_deletion = [

2286 r.server_pk

2287 for r in servercurrentrecs

2288 if r.server_pk not in server_pks_uploaded

2289 ]

2290 if server_pks_for_deletion:

2291 flag_deleted(req, batchdetails, table, server_pks_for_deletion)

2292 tablechanges.note_removal_deleted_pks(server_pks_for_deletion)

2293

2294 # Preserving all records not specifically processed above, too

2295 if batchdetails.preserving:

2296 # Preserve all, including noncurrent:

2297 preserve_all(req, batchdetails, table)

2298 # Note other preserved records, for indexing:

2299 tablechanges.note_preservation_pks(r.server_pk for r in serverrecs)

2300

2301 # (*) Indexing (and push exports)

2302 update_indexes_and_push_exports(req, batchdetails, tablechanges)

2303

2304 if DEBUG_UPLOAD:

2305 log.debug("process_table_for_onestep_upload: {}", tablechanges)

2306

2307 return tablechanges

2308

2309

2310# =============================================================================

2311# Audit functions

2312# =============================================================================

2313

2314

2315 def audit(

2316 req: "CamcopsRequest",

2317 details: str,

2318 patient_server_pk: int = None,

2319 tablename: str = None,

2320 server_pk: int = None,

2321) -> None:

2322 """

2323 Audit something.

2324 """

2325 # Add parameters and pass on:

2326 cc_audit.audit(

2327 req=req,

2328 details=details,

2329 patient_server_pk=patient_server_pk,

2330 table=tablename,

2331 server_pk=server_pk,

2332 device_id=req.tabletsession.device_id, # added

2333 remote_addr=req.remote_addr, # added

2334 user_id=req.user_id, # added

2335 from_console=False, # added

2336 from_dbclient=True, # added

2337 )

2338

2339

2340# =============================================================================

2341# Helper functions for single-user mode

2342# =============================================================================

2343

2344

2345 def make_single_user_mode_username(

2346 client_device_name: str, patient_pk: int

2347) -> str:

2348 """

2349 Returns the username for single-user mode.

2350 """

2351 return f"user-{client_device_name}-{patient_pk}"

2352

2353

2354 def json_patient_info(patient: Patient) -> str:

2355 """

2356 Converts patient details to a string representation of a JSON list (one

2357 patient) containing a single JSON dictionary (detailing that patient), with

2358 keys/formats known to the client.

2359

2360 (One item list to be consistent with patients uploaded from the tablet.)

2361

2362 Args:

2363 patient: :class:`camcops_server.cc_modules.cc_patient.Patient`

2364 """

2365 patient_dict = {

2366 TabletParam.SURNAME: patient.surname,

2367 TabletParam.FORENAME: patient.forename,

2368 TabletParam.SEX: patient.sex,

2369 TabletParam.DOB: format_datetime(

2370 patient.dob, DateFormat.ISO8601_DATE_ONLY

2371 ),

2372 TabletParam.EMAIL: patient.email,

2373 TabletParam.ADDRESS: patient.address,

2374 TabletParam.GP: patient.gp,

2375 TabletParam.OTHER: patient.other,

2376 }

2377 for idnum in patient.idnums:

2378 key = f"{TabletParam.IDNUM_PREFIX}{idnum.which_idnum}"

2379 patient_dict[key] = idnum.idnum_value # type: ignore[assignment]

2380 # One item list to be consistent with patients uploaded from the tablet

2381 return json.dumps([patient_dict])

2382

2383

2384 def get_single_server_patient(req: "CamcopsRequest") -> Patient:

2385 """

2386 Returns the patient identified by the proquint access key present in this

2387 request, or raises.

2388 """

2389 _ = req.gettext

2390

2391 patient_proquint = get_str_var(req, TabletParam.PATIENT_PROQUINT)

2392 assert patient_proquint is not None # For type checker

2393

2394 try:

2395 uuid_obj = uuid_from_proquint(patient_proquint)

2396 except InvalidProquintException:

2397 # Checksum failed or characters in wrong place

2398 # We'll do the same validation on the client so in theory

2399 # should never get here

2400 fail_user_error(

2401 _(

2402 "There is no patient with access key '{access_key}'. "

2403 "Have you entered the key correctly?"

2404 ).format(access_key=patient_proquint)

2405 )

2406

2407 server_device = Device.get_server_device(req.dbsession)

2408

2409 # noinspection PyUnboundLocalVariable,PyProtectedMember

2410 patient = (

2411 req.dbsession.query(Patient)

2412 .filter(

2413 Patient.uuid == uuid_obj,

2414 Patient._device_id == server_device.id,

2415 Patient._era == ERA_NOW,

2416 Patient._current == True, # noqa: E712

2417 )

2418 .options(joinedload(Patient.task_schedules)) # type: ignore[arg-type]

2419 .one_or_none()

2420 )

2421

2422 if patient is None:

2423 fail_user_error(

2424 _(

2425 "There is no patient with access key '{access_key}'. "

2426 "Have you entered the key correctly?"

2427 ).format(access_key=patient_proquint)

2428 )

2429

2430 if not patient.idnums:

2431 # In theory should never happen. The patient must be created with at

2432 # least one ID number. We did see this once in testing (possibly when

2433 # a patient created on a device was registered)

2434 _ = req.gettext

2435 fail_server_error(_("Patient has no ID numbers"))

2436

2437 return patient

2438

2439

2440 def get_or_create_single_user(

2441 req: "CamcopsRequest", name: str, patient: Patient

2442) -> Tuple[User, str]:

2443 """

2444 Creates a user for a patient (who's using single-user mode).

2445

2446 The user is associated (via its name) with the combination of a client

2447 device and a patient. (If a device is re-registered to another patient, the

2448 username will change.)

2449

2450 If the username already exists, then since we can't look up the password

2451 (it's irreversibly encrypted), we will set it afresh.

2452

2453 - Why is a user associated with a patient? So we can enforce that the user

2454 can upload only data relating to that patient.

2455

2456 - Why is a user associated with a device?

2457

2458 - If it is: then two users (e.g. "Device1-Bob" and "Device2-Bob") can

2459 independently work with the same patient. This will be highly

2460 confusing (mainly because it will allow "double" copies of tasks to be

2461 created, though only by manually entering things twice).

2462

2463 - If it isn't (e.g. user "Bob"): then, because registering the patient on

2464 Device2 will reset the password for the user, registering a new device

2465 for a patient will "take over" from a previous device. That has some

2466 potential for data loss if work was in progress (incomplete tasks won't

2467 be uploadable any more, and re-registering [to fix the password on the

2468 first device] would delete data).

2469

2470 - Since some confusion is better than some data loss, we associate users

2471 with a device/patient combination.

2472

2473 Args:

2474 req:

2475 a :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`

2476 name:

2477 username

2478 patient:

2479 associated :class:`camcops_server.cc_modules.cc_patient.Patient`,

2480 which also tells us the group in which to place this user

2481

2482 Returns:

2483 tuple: :class:`camcops_server.cc_modules.cc_user.User`, password

2484

2485 """

2486 dbsession = req.dbsession

2487 password = random_password()

2488 group = patient.group

2489 assert group is not None # for type checker

2490

2491 user = User.get_user_by_name(dbsession, name)

2492 creating_new_user = user is None

2493 if creating_new_user:

2494 # Create a fresh user.

2495 user = User(username=name)

2496 user.upload_group = group

2497 user.auto_generated = True

2498 user.superuser = False # should be redundant!

2499 # noinspection PyProtectedMember

2500 user.single_patient_pk = patient._pk

2501 user.set_password(req, password)

2502 if creating_new_user:

2503 dbsession.add(user)

2504 # As the username is based on a UUID, we're pretty sure another

2505 # request won't have created the same user, otherwise we'd need

2506 # to catch IntegrityError

2507 dbsession.flush()

2508

2509 membership = UserGroupMembership(user_id=user.id, group_id=group.id)

2510 membership.may_register_devices = True

2511 membership.may_upload = True

2512 user.user_group_memberships = [membership] # ... only these permissions

2513

2514 return user, password

2515

2516

2517 def random_password(length: int = 32) -> str:

2518 """

2519 Create a random password.

2520 """

2521 # Not trying anything clever with distributions of letters, digits etc

2522 characters = string.ascii_letters + string.digits + string.punctuation

2523 # We use secrets.choice() rather than random.choices() as it's better

2524 # for security/cryptography purposes.

2525 return "".join(secrets.choice(characters) for _ in range(length))

2526

2527

2528 def get_task_schedules(req: "CamcopsRequest", patient: Patient) -> str:

2529 """

2530 Gets a JSON string representation of the task schedules for a specified

2531 patient.

2532 """

2533 dbsession = req.dbsession

2534

2535 schedules = []

2536

2537 for pts in patient.task_schedules:

2538 if pts.start_datetime is None:

2539 # Minutes granularity so we are consistent with the form

2540 pts.start_datetime = req.now_utc.replace(second=0, microsecond=0)

2541 dbsession.add(pts)

2542

2543 items = []

2544

2545 for task_info in pts.get_list_of_scheduled_tasks(req):

2546 due_from = task_info.start_datetime.to_iso8601_string()

2547 due_by = task_info.end_datetime.to_iso8601_string()

2548

2549 complete = False

2550 when_completed = None

2551 task = task_info.task

2552 if task:

2553 complete = task.is_complete()

2554 if complete and task.when_last_modified:

2555 when_completed = (

2556 task.when_last_modified.to_iso8601_string()

2557 )

2558

2559 if pts.settings is not None:

2560 settings = pts.settings.get(task_info.tablename, {})

2561 else:

2562 settings = {}

2563

2564 items.append(

2565 {

2566 TabletParam.TABLE: task_info.tablename,

2567 TabletParam.ANONYMOUS: task_info.is_anonymous,

2568 TabletParam.SETTINGS: settings,

2569 TabletParam.DUE_FROM: due_from,

2570 TabletParam.DUE_BY: due_by,

2571 TabletParam.COMPLETE: complete,

2572 TabletParam.WHEN_COMPLETED: when_completed,

2573 }

2574 )

2575

2576 schedules.append(

2577 {

2578 TabletParam.TASK_SCHEDULE_NAME: pts.task_schedule.name,

2579 TabletParam.TASK_SCHEDULE_ITEMS: items,

2580 }

2581 )

2582

2583 return json.dumps(schedules)

2584

2585

2586# =============================================================================

2587# Action processors: allowed to any user

2588# =============================================================================

2589# If they return None, the framework uses the operation name as the reply in

2590# the success message. Not returning anything is the same as returning None.

2591# Authentication is performed in advance of these.

2592

2593

2594 def op_check_device_registered(req: "CamcopsRequest") -> None:

2595 """

2596 Check that a device is registered, or raise

2597 :exc:`UserErrorException`.

2598 """

2599 req.tabletsession.ensure_device_registered()

2600

2601

2602 def op_register_patient(req: "CamcopsRequest") -> Dict[str, Any]:

2603 """

2604 Registers a patient. That is, the client provides an access key. If all

2605 is well, the server returns details of that patient, as well as key

2606 server parameters, plus (if required) the username/password to use.

2607 """

2608 # -------------------------------------------------------------------------

2609 # Patient details

2610 # -------------------------------------------------------------------------

2611 patient = get_single_server_patient(req) # may fail/raise

2612 patient_info = json_patient_info(patient)

2613 reply_dict = {TabletParam.PATIENT_INFO: patient_info}

2614

2615 # -------------------------------------------------------------------------

2616 # Username/password

2617 # -------------------------------------------------------------------------

2618 client_device_name = get_str_var(req, TabletParam.DEVICE)

2619 # noinspection PyProtectedMember

2620 user_name = make_single_user_mode_username(client_device_name, patient._pk)

2621 user, password = get_or_create_single_user(req, user_name, patient)

2622 reply_dict[TabletParam.USER] = user.username

2623 reply_dict[TabletParam.PASSWORD] = password

2624

2625 # -------------------------------------------------------------------------

2626 # Intellectual property settings

2627 # -------------------------------------------------------------------------

2628 ip_use = patient.group.ip_use or IpUse()

2629 # ... if the group doesn't have an associated ip_use object, use defaults

2630 ip_dict = {

2631 TabletParam.IP_USE_COMMERCIAL: int(ip_use.commercial),

2632 TabletParam.IP_USE_CLINICAL: int(ip_use.clinical),

2633 TabletParam.IP_USE_EDUCATIONAL: int(ip_use.educational),

2634 TabletParam.IP_USE_RESEARCH: int(ip_use.research),

2635 }

2636 reply_dict[TabletParam.IP_USE_INFO] = json.dumps(ip_dict)

2637

2638 return reply_dict

2639

2640

2641# =============================================================================

2642# Action processors that require REGISTRATION privilege

2643# =============================================================================

2644

2645

2646 def op_register_device(req: "CamcopsRequest") -> Dict[str, Any]:

2647 """

2648 Register a device with the server.

2649

2650 Returns:

2651 server information dictionary (from :func:`get_server_id_info`)

2652 """

2653 dbsession = req.dbsession

2654 ts = req.tabletsession

2655 device_friendly_name = get_str_var(

2656 req, TabletParam.DEVICE_FRIENDLY_NAME, mandatory=False

2657 )

2658 # noinspection PyUnresolvedReferences

2659 device_exists = exists_in_table(

2660 dbsession, Device.__table__, Device.name == ts.device_name # type: ignore[arg-type] # noqa: E501

2661 )

2662 if device_exists:

2663 # device already registered, but accept re-registration

2664 # noinspection PyUnresolvedReferences

2665 dbsession.execute(

2666 update(Device.__table__) # type: ignore[arg-type]

2667 .where(Device.name == ts.device_name)

2668 .values(

2669 friendly_name=device_friendly_name,

2670 camcops_version=ts.tablet_version_str,

2671 registered_by_user_id=req.user_id,

2672 when_registered_utc=req.now_utc,

2673 )

2674 )

2675 else:

2676 # new registration

2677 try:

2678 # noinspection PyUnresolvedReferences

2679 dbsession.execute(

2680 Device.__table__.insert().values( # type: ignore[attr-defined]

2681 name=ts.device_name,

2682 friendly_name=device_friendly_name,

2683 camcops_version=ts.tablet_version_str,

2684 registered_by_user_id=req.user_id,

2685 when_registered_utc=req.now_utc,

2686 )

2687 )

2688 except IntegrityError:

2689 fail_user_error(INSERT_FAILED)

2690

2691 ts.reload_device()

2692 audit(

2693 req,

2694 f"register, device_id={ts.device_id}, "

2695 f"friendly_name={device_friendly_name}",

2696 tablename=Device.__tablename__,

2697 )

2698 return get_server_id_info(req)

2699

2700

2701 def op_get_extra_strings(req: "CamcopsRequest") -> Dict[str, str]:

2702 """

2703 Fetch all local extra strings from the server.

2704

2705 Returns:

2706 a SELECT-style reply (see :func:`get_select_reply`) for the

2707 extra-string table

2708 """

2709 fields = [

2710 ExtraStringFieldNames.TASK,

2711 ExtraStringFieldNames.NAME,

2712 ExtraStringFieldNames.LANGUAGE,

2713 ExtraStringFieldNames.VALUE,

2714 ]

2715 rows = req.get_all_extra_strings()

2716 reply = get_select_reply(fields, rows)

2717 audit(req, "get_extra_strings")

2718 return reply

2719

2720

2721# noinspection PyUnusedLocal

2722 def op_get_allowed_tables(req: "CamcopsRequest") -> Dict[str, str]:

2723 """

2724 Returns the names of all possible tables on the server, each paired with

2725 the minimum client (tablet) version that will be accepted for each table.

2726 (Typically these are all the same as the minimum global tablet version.)

2727

2728 Uses the SELECT-like syntax (see :func:`get_select_reply`).

2729 """

2730 tables_versions = all_tables_with_min_client_version()

2731 fields = [

2732 AllowedTablesFieldNames.TABLENAME,

2733 AllowedTablesFieldNames.MIN_CLIENT_VERSION,

2734 ]

2735 rows = [[k, str(v)] for k, v in tables_versions.items()]

2736 reply = get_select_reply(fields, rows)

2737 audit(req, "get_allowed_tables")

2738 return reply

2739

2740

2741 def op_get_task_schedules(req: "CamcopsRequest") -> Dict[str, str]:

2742 """

2743 Return details of the task schedules for the patient associated with

2744 this request, for single-user mode. Also returns details of the single

2745 patient, in case that's changed.

2746 """

2747 patient = get_single_server_patient(req)

2748 patient_info = json_patient_info(patient)

2749 task_schedules = get_task_schedules(req, patient)

2750 return {

2751 TabletParam.PATIENT_INFO: patient_info,

2752 TabletParam.TASK_SCHEDULES: task_schedules,

2753 }

2754

2755

2756# =============================================================================

2757# Action processors that require UPLOAD privilege

2758# =============================================================================

2759

2760

2761# noinspection PyUnusedLocal

2762 def op_check_upload_user_and_device(req: "CamcopsRequest") -> None:

2763 """

2764 Stub function for the operation to check that a user is valid.

2765

2766 To get this far, the user has to be valid, so this function doesn't

2767 actually have to do anything.

2768 """

2769 pass # don't need to do anything!

2770

2771

2772# noinspection PyUnusedLocal

2773 def op_get_id_info(req: "CamcopsRequest") -> Dict[str, Any]:

2774 """

2775 Fetch server ID information; see :func:`get_server_id_info`.

2776 """

2777 return get_server_id_info(req)

2778

2779

2780 def op_start_upload(req: "CamcopsRequest") -> None:

2781 """

2782 Begin an upload.

2783 """

2784 start_device_upload_batch(req)

2785

2786

2787 def op_end_upload(req: "CamcopsRequest") -> None:

2788 """

2789 Ends an upload and commits changes.

2790 """

2791 batchdetails = get_batch_details(req)

2792 # ensure it's the same user finishing as starting!

2793 end_device_upload_batch(req, batchdetails)

2794

2795

2796 def op_upload_table(req: "CamcopsRequest") -> str:

2797 """

2798 Upload a table.

2799

2800 Incoming information in the POST request includes a CSV list of fields, a

2801 count of the number of records being provided, and a set of variables named

2802 ``record0`` ... ``record{nrecords - 1}``, each containing a CSV list of

2803 SQL-encoded values.

2804

2805 Typically used for smaller tables, i.e. most except for BLOBs.

2806 """

2807 table = get_table_from_req(req, TabletParam.TABLE)

2808

2809 allowed_nonexistent_fields = [] # type: List[str]

2810 # noinspection PyUnresolvedReferences

2811 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__:

2812 for x in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):

2813 allowed_nonexistent_fields.extend(

2814 [

2815 FP_ID_NUM + str(x),

2816 FP_ID_DESC + str(x),

2817 FP_ID_SHORT_DESC + str(x),

2818 ]

2819 )

2820

2821 fields = get_fields_from_post_var(

2822 req,

2823 table,

2824 TabletParam.FIELDS,

2825 allowed_nonexistent_fields=allowed_nonexistent_fields,

2826 )

2827 nrecords = get_int_var(req, TabletParam.NRECORDS)

2828

2829 nfields = len(fields)

2830 if nfields < 1:

2831 # May never be reached as the POST var can't be empty

2832 fail_user_error(

2833 f"{TabletParam.FIELDS}={nfields}: can't be less than 1"

2834 )

2835 if nrecords < 0:

2836 fail_user_error(

2837 f"{TabletParam.NRECORDS}={nrecords}: can't be less than 0"

2838 )

2839

2840 batchdetails = get_batch_details(req)

2841

2842 ts = req.tabletsession

2843 if ts.explicit_pkname_for_upload_table: # q.v.

2844 # New client: tells us the PK name explicitly.

2845 clientpk_name = get_single_field_from_post_var(

2846 req, table, TabletParam.PKNAME

2847 )

2848 else:

2849 # Old client. Either (a) old Titanium client, in which the client PK

2850 # is in fields[0] [SUCH CLIENTS ARE NO LONGER SUPPORTED AS OF SERVER

2851 # v2.4.15], or (b) an early C++ client, in which there was no

2852 # guaranteed order (and no explicit PK name was sent). However, in

2853 # either case, the client PK name was (is) always "id".

2854 clientpk_name = TABLET_ID_FIELD

2855 ensure_valid_field_name(table, clientpk_name)

2856 server_pks_uploaded = [] # type: List[int]

2857 n_new = 0

2858 n_modified = 0

2859 n_identical = 0

2860 dirty = False

2861 serverrecs = get_server_live_records(

2862 req,

2863 ts.device_id,

2864 table,

2865 clientpk_name=clientpk_name,

2866 current_only=True,

2867 )

2868 for r in range(nrecords):

2869 recname = TabletParam.RECORD_PREFIX + str(r)

2870 values = get_values_from_post_var(req, recname)

2871 nvalues = len(values)

2872 if nvalues != nfields:

2873 errmsg = (

2874 f"Number of fields in field list ({nfields}) doesn't match "

2875 f"number of values in record {r} ({nvalues})"

2876 )

2877 log.warning(errmsg + f"\nfields: {fields!r}\nvalues: {values!r}")

2878 fail_user_error(errmsg)

2879 valuedict = dict(zip(fields, values))

2880 # log.debug("table {!r}, record {}: {!r}", table.name, r, valuedict)

2881 # CORE: CALLS upload_record_core

2882 urr = upload_record_core(

2883 req,

2884 batchdetails,

2885 table,

2886 clientpk_name,

2887 valuedict,

2888 server_live_current_records=serverrecs,

2889 )

2890 if urr.oldserverpk is not None: # was an existing record

2891 server_pks_uploaded.append(urr.oldserverpk)

2892 if urr.newserverpk is None:

2893 n_identical += 1

2894 else:

2895 n_modified += 1

2896 else: # entirely new

2897 n_new += 1

2898 if urr.dirty:

2899 dirty = True

2900

2901 # Now deal with any ABSENT (not in uploaded data set) conditions.

2902 server_pks_for_deletion = [

2903 r.server_pk

2904 for r in serverrecs

2905 if r.server_pk not in server_pks_uploaded

2906 ]

2907 # Note that "deletion" means "end of the line"; records that are modified

2908 # and replaced were handled by upload_record_core().

2909 n_deleted = len(server_pks_for_deletion)

2910 if n_deleted > 0:

2911 flag_deleted(req, batchdetails, table, server_pks_for_deletion)

2912

2913 # Set dirty/clean status

2914 if (

2915 dirty

2916 or n_new > 0

2917 or n_modified > 0

2918 or n_deleted > 0

2919 or any(sr.move_off_tablet for sr in serverrecs)

2920 ):

2921 # ... checks on n_new and n_modified are redundant; dirty will be True

2922 mark_table_dirty(req, table)

2923 elif batchdetails.preserving and not serverrecs:

2924 # We've scanned this table, and there would be no work to do to

2925 # preserve records from previous uploads.

2926 mark_table_clean(req, table)

2927

2928 # Special for old tablets:

2929 # noinspection PyUnresolvedReferences

2930 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__:

2931 # noinspection PyUnresolvedReferences

2932 mark_table_dirty(req, PatientIdNum.__table__) # type: ignore[arg-type]

2933 # Mark patient ID numbers for deletion if their parent Patient is

2934 # similarly being marked for deletion

2935 # noinspection PyUnresolvedReferences,PyProtectedMember

2936 req.dbsession.execute(

2937 update(PatientIdNum.__table__) # type: ignore[arg-type]

2938 .where(PatientIdNum._device_id == Patient._device_id)

2939 .where(PatientIdNum._era == ERA_NOW)

2940 .where(PatientIdNum.patient_id == Patient.id)

2941 .where(Patient._pk.in_(server_pks_for_deletion))

2942 .where(Patient._era == ERA_NOW) # shouldn't be in doubt!

2943 .values(_removal_pending=1, _successor_pk=None)

2944 )

2945

2946 # Auditing occurs at commit_all.

2947 log.info(

2948 "Upload successful; {n} records uploaded to table {t} "

2949 "({new} new, {mod} modified, {i} identical, {nd} deleted)",

2950 n=nrecords,

2951 t=table.name,

2952 new=n_new,

2953 mod=n_modified,

2954 i=n_identical,

2955 nd=n_deleted,

2956 )

2957 return f"Table {table.name} upload successful"

2958

2959

2960 def op_upload_record(req: "CamcopsRequest") -> str:

2961 """

2962 Upload an individual record. (Typically used for BLOBs.)

2963 Incoming POST information includes a CSV list of fields and a CSV list of

2964 values.

2965 """

2966 batchdetails = get_batch_details(req)

2967 table = get_table_from_req(req, TabletParam.TABLE)

2968 clientpk_name = get_single_field_from_post_var(

2969 req, table, TabletParam.PKNAME

2970 )

2971 valuedict = get_fields_and_values(

2972 req, table, TabletParam.FIELDS, TabletParam.VALUES

2973 )

2974 urr = upload_record_core(

2975 req, batchdetails, table, clientpk_name, valuedict

2976 )

2977 if urr.dirty:

2978 mark_table_dirty(req, table)

2979 if urr.oldserverpk is None:

2980 log.info("upload-insert")

2981 return "UPLOAD-INSERT"

2982 else:

2983 if urr.newserverpk is None:

2984 log.info("upload-update: skipping existing record")

2985 else:

2986 log.info("upload-update")

2987 return "UPLOAD-UPDATE"

2988 # Auditing occurs at commit_all.

2989

2990

2991 def op_upload_empty_tables(req: "CamcopsRequest") -> str:

2992 """

2993 The tablet supplies a list of tables that are empty at its end, and we

2994 will 'wipe' all appropriate tables; this reduces the number of HTTP

2995 requests.

2996 """

2997 tables = get_tables_from_post_var(req, TabletParam.TABLES)

2998 batchdetails = get_batch_details(req)

2999 to_dirty = [] # type: List[Table]

3000 to_clean = [] # type: List[Table]

3001 for table in tables:

3002 nrows_affected = flag_all_records_deleted(req, table)

3003 if nrows_affected > 0:

3004 to_dirty.append(table)

3005 elif batchdetails.preserving:

3006 # There are no records in the current era for this device.

3007 to_clean.append(table)

3008 # In the fewest number of queries:

3009 mark_tables_dirty(req, to_dirty)

3010 mark_tables_clean(req, to_clean)

3011 log.info("upload_empty_tables")

3012 # Auditing occurs at commit_all.

3013 return "UPLOAD-EMPTY-TABLES"

3014

3015

3016 def op_start_preservation(req: "CamcopsRequest") -> str:

3017 """

3018 Marks this upload batch as one in which all records will be preserved

3019 (i.e. moved from NOW-era to an older era, so they can be deleted safely

3020 from the tablet).

3021

3022 Without this, individual records can still be marked for preservation if

3023 their MOVE_OFF_TABLET_FIELD field (``_move_off_tablet``) is set; see

3024 :func:`upload_record` and the functions it calls.

3025 """

3026 get_batch_details(req)

3027 start_preserving(req)

3028 log.info("start_preservation successful")

3029 # Auditing occurs at commit_all.

3030 return "STARTPRESERVATION"

3031

3032

3033 def op_delete_where_key_not(req: "CamcopsRequest") -> str:

3034 """

3035 Marks records for deletion, for a device/table, where the client PK

3036 is not in a specified list.

3037 """

3038 table = get_table_from_req(req, TabletParam.TABLE)

3039 clientpk_name = get_single_field_from_post_var(

3040 req, table, TabletParam.PKNAME

3041 )

3042 clientpk_values = get_values_from_post_var(req, TabletParam.PKVALUES)

3043

3044 get_batch_details(req)

3045 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values)

3046 # Auditing occurs at commit_all.

3047 # log.info("delete_where_key_not successful; table {} trimmed", table)

3048 return "Trimmed"

3049

3050

3051 def op_which_keys_to_send(req: "CamcopsRequest") -> str:

3052 """

3053 Intended use: "For my device, and a specified table, here are my

3054 client-side PKs (as a CSV list), and the modification dates for each

3055 corresponding record (as a CSV list). Please tell me which records have

3056 mismatching dates on the server, i.e. those that I need to re-upload."

3057

3058 Used particularly for BLOBs, to reduce traffic, i.e. so we don't have to

3059 send a lot of BLOBs.

3060

3061 Note new ``TabletParam.MOVE_OFF_TABLET_VALUES`` parameter in server v2.3.0,

3062 with bugfix for pre-2.3.0 clients that won't send this; see changelog.

3063 """

3064 # -------------------------------------------------------------------------

3065 # Get details

3066 # -------------------------------------------------------------------------

3067 table = get_table_from_req(req, TabletParam.TABLE)

3068 clientpk_name = get_single_field_from_post_var(

3069 req, table, TabletParam.PKNAME

3070 )

3071 clientpk_values = get_values_from_post_var(

3072 req, TabletParam.PKVALUES, mandatory=False

3073 )

3074 # ... should be autoconverted to int, but we check below

3075 client_dates = get_values_from_post_var(

3076 req, TabletParam.DATEVALUES, mandatory=False

3077 )

3078 # ... will be in string format

3079

3080 npkvalues = len(clientpk_values)

3081 ndatevalues = len(client_dates)

3082 if npkvalues != ndatevalues:

3083 fail_user_error(

3084 f"Number of PK values ({npkvalues}) doesn't match number of dates "

3085 f"({ndatevalues})"

3086 )

3087

3088 # v2.3.0:

3089 move_off_tablet_values = [] # type: List[int] # for type checker

3090 if req.has_param(TabletParam.MOVE_OFF_TABLET_VALUES):

3091 client_reports_move_off_tablet = True

3092 move_off_tablet_values = get_values_from_post_var(

3093 req, TabletParam.MOVE_OFF_TABLET_VALUES, mandatory=True

3094 )

3095 # ... should be autoconverted to int

3096 n_motv = len(move_off_tablet_values)

3097 if n_motv != npkvalues:

3098 fail_user_error(

3099 f"Number of move-off-tablet values ({n_motv}) doesn't match "

3100 f"number of PKs ({npkvalues})"

3101 )

3102 try:

3103 move_off_tablet_values = [bool(x) for x in move_off_tablet_values]

3104 except (TypeError, ValueError):

3105 # Probably never reached given pretty much anything standard can be

3106 # converted to bool

3107 fail_user_error(

3108 f"Bad move-off-tablet values: {move_off_tablet_values!r}"

3109 )

3110 else:

3111 client_reports_move_off_tablet = False

3112 log.warning(

3113 "op_which_keys_to_send: old client not reporting "

3114 "{}; requesting all records",

3115 TabletParam.MOVE_OFF_TABLET_VALUES,

3116 )

3117

3118 clientinfo = [] # type: List[WhichKeyToSendInfo]

3119

3120 for i in range(npkvalues):

3121 cpkv = clientpk_values[i]

3122 if not isinstance(cpkv, int):

3123 fail_user_error(f"Bad (non-integer) client PK: {cpkv!r}")

3124 try:

3125 dt = coerce_to_pendulum(client_dates[i])

3126 if dt is None:

3127 fail_user_error(f"Missing date/time for client PK {cpkv}")

3128 except ValueError:

3129 fail_user_error(f"Bad date/time: {client_dates[i]!r}")

3130 clientinfo.append(

3131 WhichKeyToSendInfo(

3132 client_pk=cpkv,

3133 client_when=dt,

3134 client_move_off_tablet=(

3135 move_off_tablet_values[i] # type: ignore[arg-type]

3136 if client_reports_move_off_tablet

3137 else False

3138 ),

3139 )

3140 )

3141

3142 # -------------------------------------------------------------------------

3143 # Work out the answer

3144 # -------------------------------------------------------------------------

3145 batchdetails = get_batch_details(req)

3146

3147 # 1. The client sends us all its PKs. So "delete" anything not in that

3148 # list.

3149 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values)

3150

3151 # 2. See which ones are new or updates.

3152 client_pks_needed = [] # type: List[int]

3153 client_pk_to_serverrec = client_pks_that_exist(

3154 req, table, clientpk_name, clientpk_values

3155 )

3156 for wk in clientinfo:

3157 if client_reports_move_off_tablet:

3158 if wk.client_pk not in client_pk_to_serverrec:

3159 # New on the client; we want it

3160 client_pks_needed.append(wk.client_pk)

3161 else:

3162 # We know about some version of this client record.

3163 serverrec = client_pk_to_serverrec[wk.client_pk]

3164 if serverrec.server_when != wk.client_when:

3165 # Modified on the client; we want it

3166 client_pks_needed.append(wk.client_pk)

3167 elif serverrec.move_off_tablet != wk.client_move_off_tablet:

3168 # Not modified on the client. But it is being preserved.

3169 # We don't need to ask the client for it again, but we do

3170 # need to mark the preservation.

3171 flag_record_for_preservation(

3172 req, batchdetails, table, serverrec.server_pk

3173 )

3174

3175 else:

3176 # Client hasn't told us about the _move_off_tablet flag. Always

3177 # request the record (workaround potential bug in old clients).

3178 client_pks_needed.append(wk.client_pk)

3179

3180 # Success

3181 pk_csv_list = ",".join(

3182 [str(x) for x in client_pks_needed if x is not None]

3183 )

3184 # log.info("which_keys_to_send successful: table {}", table.name)

3185 return pk_csv_list

3186

3187

3188 def op_validate_patients(req: "CamcopsRequest") -> str:

3189 """

3190 As of v2.3.0, the client can use this command to validate patients against

3191 arbitrary server criteria -- definitely the upload/finalize ID policies,

3192 but potentially also other criteria of the server's (like matching against

3193 a bank of predefined patients).

3194

3195 Compare ``NetworkManager::getPatientInfoJson()`` on the client.

3196

3197 There is a slight weakness with respect to "single-patient" users, in that

3198 the client *asks* if the patients are OK (rather than the server

3199 *enforcing* that they are OK, via hooks into :func:`op_upload_table`,

3200 :func:`op_upload_record`, :func:`op_upload_entire_database` -- made more

3201 complex because ID numbers are not uploaded to the same table...). In

3202 principle, the weakness is that a user could (a) crack their assigned

3203 password and (b) rework the CamCOPS client, in order to upload "bad"

3204 patient data into their assigned group.

3205

3206 todo:

3207 address this by having the server *require* patient validation for

3208 all uploads?

3209

3210 """

3211 pt_json_list = get_json_from_post_var(

3212 req,

3213 TabletParam.PATIENT_INFO,

3214 decoder=PATIENT_INFO_JSON_DECODER,

3215 mandatory=True,

3216 )

3217 if not isinstance(pt_json_list, list):

3218 fail_user_error("Top-level JSON is not a list")

3219 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id)

3220 for pt_dict in pt_json_list:

3221 ensure_valid_patient_json(req, group, pt_dict)

3222 return SUCCESS_MSG

3223

3224

3225 def op_upload_entire_database(req: "CamcopsRequest") -> str:

3226 """

3227 Perform a one-step upload of the entire database.

3228

3229 - From v2.3.0.

3230 - Therefore, we do not have to cope with old-style ID numbers.

3231 """

3232 # Roll back and clear any outstanding changes

3233 clear_device_upload_batch(req)

3234

3235 # Fetch the JSON, with sanity checks

3236 preserving = get_bool_int_var(req, TabletParam.FINALIZING)

3237 pknameinfo = get_json_from_post_var(

3238 req, TabletParam.PKNAMEINFO, decoder=DB_JSON_DECODER, mandatory=True

3239 )

3240 if not isinstance(pknameinfo, dict):

3241 fail_user_error("PK name info JSON is not a dict")

3242 dbdata = get_json_from_post_var(

3243 req, TabletParam.DBDATA, decoder=DB_JSON_DECODER, mandatory=True

3244 )

3245 if not isinstance(dbdata, dict):

3246 fail_user_error("Database data JSON is not a dict")

3247

3248 # Sanity checks

3249 dbdata_tablenames = sorted(dbdata.keys())

3250 pkinfo_tablenames = sorted(pknameinfo.keys())

3251 if pkinfo_tablenames != dbdata_tablenames:

3252 fail_user_error("Table names don't match from (1) DB data (2) PK info")

3253 duff_tablenames = sorted(

3254 list(set(dbdata_tablenames) - set(CLIENT_TABLE_MAP.keys()))

3255 )

3256 if duff_tablenames:

3257 fail_user_error(

3258 f"Attempt to upload nonexistent tables: {duff_tablenames!r}"

3259 )

3260

3261 # Perform the upload

3262 batchdetails = BatchDetails(

3263 req.now_utc, preserving=preserving, onestep=True

3264 ) # NB special "onestep" option

3265 # Process the tables in a certain order:

3266 tables = sorted(CLIENT_TABLE_MAP.values(), key=upload_commit_order_sorter)

3267 changelist = [] # type: List[UploadTableChanges]

3268 for table in tables:

3269 clientpk_name = pknameinfo.get(table.name, "")

3270 rows = dbdata.get(table.name, [])

3271 tablechanges = process_table_for_onestep_upload(

3272 req, batchdetails, table, clientpk_name, rows

3273 )

3274 changelist.append(tablechanges)

3275

3276 # Audit

3277 audit_upload(req, changelist)

3278

3279 # Done

3280 return SUCCESS_MSG

3281

3282

3283# =============================================================================

3284# Action maps

3285# =============================================================================

3286

3287

3288 class Operations:

3289 """

3290 Constants giving the name of operations (commands) accepted by this API.

3291 """

3292

3293 CHECK_DEVICE_REGISTERED = "check_device_registered"

3294 CHECK_UPLOAD_USER_DEVICE = "check_upload_user_and_device"

3295 DELETE_WHERE_KEY_NOT = "delete_where_key_not"

3296 END_UPLOAD = "end_upload"

3297 GET_ALLOWED_TABLES = "get_allowed_tables" # v2.2.0

3298 GET_EXTRA_STRINGS = "get_extra_strings"

3299 GET_ID_INFO = "get_id_info"

3300 GET_TASK_SCHEDULES = "get_task_schedules" # v2.4.0

3301 REGISTER = "register"

3302 REGISTER_PATIENT = "register_patient" # v2.4.0

3303 START_PRESERVATION = "start_preservation"

3304 START_UPLOAD = "start_upload"

3305 UPLOAD_EMPTY_TABLES = "upload_empty_tables"

3306 UPLOAD_ENTIRE_DATABASE = "upload_entire_database" # v2.3.0

3307 UPLOAD_RECORD = "upload_record"

3308 UPLOAD_TABLE = "upload_table"

3309 VALIDATE_PATIENTS = "validate_patients" # v2.3.0

3310 WHICH_KEYS_TO_SEND = "which_keys_to_send"

3311

3312

3313 OPERATIONS_ANYONE = {

3314 Operations.CHECK_DEVICE_REGISTERED: op_check_device_registered,

3315 # Anyone can register a patient provided they have the right unique code

3316 Operations.REGISTER_PATIENT: op_register_patient,

3317}

3318 OPERATIONS_REGISTRATION = {

3319 Operations.GET_ALLOWED_TABLES: op_get_allowed_tables, # v2.2.0

3320 Operations.GET_EXTRA_STRINGS: op_get_extra_strings,

3321 Operations.GET_TASK_SCHEDULES: op_get_task_schedules,

3322 Operations.REGISTER: op_register_device,

3323}

3324 OPERATIONS_UPLOAD = {

3325 Operations.CHECK_UPLOAD_USER_DEVICE: op_check_upload_user_and_device,

3326 Operations.DELETE_WHERE_KEY_NOT: op_delete_where_key_not,

3327 Operations.END_UPLOAD: op_end_upload,

3328 Operations.GET_ID_INFO: op_get_id_info,

3329 Operations.START_PRESERVATION: op_start_preservation,

3330 Operations.START_UPLOAD: op_start_upload,

3331 Operations.UPLOAD_EMPTY_TABLES: op_upload_empty_tables,

3332 Operations.UPLOAD_ENTIRE_DATABASE: op_upload_entire_database,

3333 Operations.UPLOAD_RECORD: op_upload_record,

3334 Operations.UPLOAD_TABLE: op_upload_table,

3335 Operations.VALIDATE_PATIENTS: op_validate_patients, # v2.3.0

3336 Operations.WHICH_KEYS_TO_SEND: op_which_keys_to_send,

3337}

3338

3339

3340# =============================================================================

3341# Client API main functions

3342# =============================================================================

3343

3344

3345 def main_client_api(req: "CamcopsRequest") -> Dict[str, str]:

3346 """

3347 Main HTTP processor.

3348

3349 For success, returns a dictionary to send (will use status '200 OK')

3350 For failure, raises an exception.

3351 """

3352 # log.info("CamCOPS database script starting at {}",

3353 # format_datetime(req.now, DateFormat.ISO8601))

3354 ts = req.tabletsession

3355 fn = None

3356

3357 if ts.operation in OPERATIONS_ANYONE:

3358 fn = OPERATIONS_ANYONE.get(ts.operation)

3359

3360 elif ts.operation in OPERATIONS_REGISTRATION:

3361 ts.ensure_valid_user_for_device_registration()

3362 fn = OPERATIONS_REGISTRATION.get(ts.operation)

3363

3364 elif ts.operation in OPERATIONS_UPLOAD:

3365 ts.ensure_valid_device_and_user_for_uploading()

3366 fn = OPERATIONS_UPLOAD.get(ts.operation) # type: ignore[assignment]

3367

3368 if not fn:

3369 fail_unsupported_operation(ts.operation)

3370 result = fn(req)

3371 if result is None:

3372 # generic success

3373 result = {TabletParam.RESULT: ts.operation}

3374 elif not isinstance(result, dict):

3375 # convert strings (etc.) to a dictionary

3376 result = {TabletParam.RESULT: result}

3377 return result

3378

3379

3380@view_config(

3381 route_name=Routes.CLIENT_API,

3382 request_method=HttpMethod.POST,

3383 permission=NO_PERMISSION_REQUIRED,

3384)

3385@view_config(

3386 route_name=Routes.CLIENT_API_ALIAS,

3387 request_method=HttpMethod.POST,

3388 permission=NO_PERMISSION_REQUIRED,

3389)

3390 def client_api(req: "CamcopsRequest") -> Response:

3391 """

3392 View for client API. All tablet interaction comes through here.

3393 Wraps :func:`main_client_api`. Handles exceptions.

3394

3395 Internally, replies are managed as dictionaries.

3396 For the final reply, the dictionary is converted to text in this format:

3397

3398 .. code-block:: none

3399

3400 k1:v1

3401 k2:v2

3402 k3:v3

3403 ...

3404 """

3405 # log.debug("{!r}", req.environ)

3406 # log.debug("{!r}", req.params)

3407 t0 = time.time() # in seconds

3408

3409 # -------------------------------------------------------------------------

3410 # Establish session (requires something coherent from the client)

3411 # -------------------------------------------------------------------------

3412 try:

3413 ts = req.tabletsession

3414 except UserErrorException as e:

3415 log.warning("CLIENT-SIDE SCRIPT ERROR: {}", e)

3416 return TextResponse(

3417 "Not a valid CamCOPS API request\n", status="400 Bad Request"

3418 )

3419

3420 # -------------------------------------------------------------------------

3421 # Call main API

3422 # -------------------------------------------------------------------------

3423 try:

3424 resultdict = main_client_api(req)

3425 resultdict[TabletParam.SUCCESS] = SUCCESS_CODE

3426 status = "200 OK"

3427

3428 except UserErrorException as e:

3429 log.warning("CLIENT-SIDE SCRIPT ERROR: {}", e)

3430 resultdict = {

3431 TabletParam.SUCCESS: FAILURE_CODE,

3432 TabletParam.ERROR: escape_newlines(str(e)),

3433 }

3434 status = "200 OK"

3435

3436 except ServerErrorException as e:

3437 log.error("SERVER-SIDE SCRIPT ERROR: {}", e)

3438 # rollback? Not sure

3439 resultdict = {

3440 TabletParam.SUCCESS: FAILURE_CODE,

3441 TabletParam.ERROR: escape_newlines(str(e)),

3442 }

3443 status = "503 Database Unavailable: " + str(e)

3444

3445 except Exception as e:

3446 # All other exceptions. May include database write failures.

3447 # Let's return with status '200 OK'; though this seems dumb, it means

3448 # the tablet user will at least see the message.

3449 log.exception("Unhandled exception") # + traceback.format_exc()

3450 resultdict = {

3451 TabletParam.SUCCESS: FAILURE_CODE,

3452 TabletParam.ERROR: escape_newlines(exception_description(e)),

3453 }

3454 status = "200 OK"

3455

3456 # Add session token information

3457 resultdict[TabletParam.SESSION_ID] = ts.session_id

3458 resultdict[TabletParam.SESSION_TOKEN] = ts.session_token

3459

3460 # Convert dictionary to text in name-value pair format

3461 txt = "".join(f"{k}:{v}\n" for k, v in resultdict.items())

3462

3463 t1 = time.time()

3464 log.debug("Time in script (s): {t}", t=t1 - t0)

3465

3466 return TextResponse(txt, status=status)