Pootle 2.8 release notes — Pootle 2.9.0b4 documentation

# Pootle 2.8 release notes

Released on 16 August 2017

Welcome to Pootle 2.8!

If you want to try it, check one of the following:

- Installation instructions
- Upgrading instructions

## Changes in Requirements

- Django>=1.10.5,<1.11
- zlib1g-dev required for lxml
- MySQL support uses mysqlclient instead of MySQLdb
- django-transaction-hooks is no longer used

## Major Changes

- Performed a security audit. Made various changes to close potential security vectors including DOS type atacks.
- Browse view optimisation. The performance of the brose mode now scales well for large projects. Statistics are now in real time.
- Improved editor performance. Retrieval of editable units now correctly retrieves pages and prevents massive database load.
- Timeline data. The timeline data has been migrated and improved to store more data changes and clean up past data. This is a step to providing more detailed change history and is needed to do unit level comparisons for Pootle FS.
- Statistics generation and display. We no longer use rqworkers to calculate statistics. Instead these are live and always correct being updated on unit changes. The system is faster then previous statistics calculations.
- Pootle FS (beta) - provides bidirectional version control synchronisation.

## Details of changes

Below we provide much more detail. These are by no means exhaustive, view the git log for complete information.

- MySQL now uses the `mysqlclient` database driver instead of `MySQLdb`, this is Django’s preferred database driver.
- Performed a security audit
- JavaScript fixes addressing performance and memory leaks in the editor
- Improved editor performance
- Extensive review and fixes to RTL layout
- Refactored code for syncing stores - as part of the internal rework of core functionality in Pootle we reworked synchronisation code. This will eventually be dropped when we move fully onto Pootle FS.
- Changed the default `robots.txt`. It is now a static file which you can adjust for your site. The site is `allow` by default.
- Editor:
  - Suggestions:
    - Support rejecting/accepting suggestions with comments
      - Suggesters are emailed based on new `POOTLE_EMAIL_FEEDBACK_ENABLED` setting
    - Accept suggestion without `can review` permissions if submitted translation is 100% match.
    - Users cannot submit suggestions that match a previously submitted suggestion or the current translation.
  - Current unit is shown at the top of the unit list with one context row, this makes it is easier to use the navigation bar and prevents UI jumpiness.
  - Faster terminology matching yielding more results by using stemming.
  - Special non-display characters are depicted by symbols from a custom-built font:
    - Properly display whitespace as special character e.g. non breaking space, newline, tab.
    - Technical details of escape sequences are omitted in the output displayed to end users, making it easier to enter correct data.
  - Buttons to add language specific special characters display helpful information in tooltips.
  - Errors fixed:
    - Fixed bugs in muting/unmuting checks
    - Fixed filtering translations by month
    - Fixed `TypeError` error when filter in editor gets no units
    - Fixed bug that prevented translators from clicking on context rows to edit those context units
  - Translation memory:
    - Display original and translations side-by-side for TM results
    - AmaGama Translation Memory is now queried using CORS
    - AmaGama is only queried if the source language is supported by amaGama
  - Perform all highlighting in the client.
  - Clearly present plurals to translator.
  - Allow text from the similar translations area to be selected.
  - Force word wrapping on long strings with no spaces.
  - Improvements on timeline.
  - Cross-language translation is now restricted to admins. It heavily impacts performance and translators are unlikely to require it while admins may have a valid reason to use it.
  - Removed ability to clear language or project dropdown to prevent performance degradation. This prevents users inadvertantly hitting very expensive queries.
  - Alternate source language translations are no longer displayed for anonymous users to prevent performance costs for users who aren’t able to translate.
  - Editor is disabled for users without the required permissions. Reviewing suggestions is similarly disabled without required permissions.
  - Check categories can now be used in dropdown to filter units. You can review all ‘critical’ check failures at once.
  - Auto-matched translations are now highlighted to indicate that they came from translation memory to prevent confusing users.
  - Incomplete plural translations may now be submitted.
  - String error reporting form is now clearer and prevents empty reports from being submitted.
- Added suggestion bulk management:
  - Provides filtering by user, etc.
  - Allows to review multiple suggestions at once and reject/accept them at once optionally providing a comment for the suggesters
- Configuration system - a generic system to store configuration information for Pootle.
- Plugin framework - allowing Pootle to use plugins to expand its functionality.
- Comment system
- Removed Plurr format checks
- Removed ENChecker
- Added support to have several formats in the same project
- Browse pages:
  - Refactored stats backend:
    - We now store statistics in the database and have removed the need for rqworkers to calculate stats.
    - The stats refresh notice has been removed as all stats are now up-to-date, always.
    - Faster stats retrieval is now possible as stats are always up-to-date and we can get it directly from the database.
  - Disabled items are hidden by default but admin users can select to shown them.
  - Changed order of columns to highlight latest activity and pending work:
    - Last updated data is now only shown to admin users
  - Altered order in which some items are listed by default:
    - Projects and languages are sorted by most recent translators changes to highlight activity
    - Virtual folders are sorted by priority to highlight most important strings to translate
  - Hid most of the special ‘templates’ language data as it is unnecessary and can be confusing.
  - Got rid of fat cookies:
    - Increases responsiveness and removes security issue
    - Most data is now stored in user session instead
    - Sidebar is no longer automatically open for anonymous users when an announcement changes.
  - Leaderboard on top panel and expanded stats panel:
    - The top panel will show the three users with the highest score.
    - Expanded stats shows the top contributors scores and other detailed information about current location.
  - Numbers are rendered in a locale aware fashion.
  - Improvements to the statistics table for overly long filenames and smaller screens.
  - Files dropdown no longer keeps references to empty directories.
  - Fixed issue where the “Back” button would sometimes not work.
  - Fixed issue with project dropdown when there are projects without a name.
- Search:
  - No longer autocompletes
  - Added pluggable search backend
  - Search widget is disabled if user cannot translate. This is to prevent any load on the server for users who are not able to contribute.
  - Old ‘Exact Match’ was separated into ‘Case-sensitive match’ and ‘Phrase match’ allowing finer-grained searches and removing the previous confusion about the actual intent of the options.
- Added team page:
  - Only for languages so far, and only available to language managers
  - Replaces permissions with roles
  - Provides direct access to suggestion bulk management
- Revamped user profile page. The aim is to slowly draw more information onto this page and make it a hub for translators.
- Removed for performance reasons:
  - Removed statistics from user profiles. Will be brought back in the future.
  - Removed export view. This has been replaced with TMX export functionality, download still remains.
  - Removed performance hogging “More stats” in admin dashboard. While it has some useful information there are better ways to get this data.
- Removed reports feature. This was a potential security area and data leak. We will bring this back now that we have finer grained change tracking.
- Pootle’s own localization changes:
  - Updated translations. You can still contribute translation updates for your language.
  - Now compilejsi18n is used to compile JavaScript translations into assets, thus requiring django-statici18n app.
  - Password reset email is now localizable in Pootle.
  - Multiple changes in localizable strings to ease translation.
  - Select2 localization is bundled to ensure Select2 is shown localized.
- Upload and download:
  - Disabled upload for non-PO projects as conflict handling currently only works in PO.
  - Admins can upload translations as other user allowing correct crediting for translations.
  - Fixed error for stores with no revision.
  - Added the ability to download TMX exports.
- New Machine Translation providers:
  - Caighdeán - Irish
  - Welsh
- Refactoring of models to increase performance, including dropping unnecessary indices.
- User input is sanitized for outgoing emails
- Usernames using latin1 characters are now allowed
- Improved RQ usage and new management commands
- Changed Pootle logo and styling
- Added the ability to use a custom logo with `POOTLE_CUSTOM_LOGO`
- Documentation updates

### Pootle FS (beta)

Pootle FS enables synchronization of Pootle against a filesystem, or version control system, handling conflict resolution and other situations of two files being out of sync.

Pootle FS follows a git like command execution. We’ve designed it such that we expect there to be no data loss when conflicts are discovered. Any conflicts are turned into suggestions which can be resolved in Pootle.

Pootle FS is still in beta as we’d like to make sure that all the bugs are washed out before making it an official and default part of Pootle.

sync_stores and update_stores are still the default method of interacting with Pootle. We expect these to remain for some time, but expect the next version of Pootle to use to Pootle FS infrastructure to manage and handle these commands.

- Added admin UI to set up projects configuration and language mapping
- CLI - adds `info`, `fetch`, `resolve`, `sync`, `add` and `rm` commands
- LanguageMapper - allows differing codes on the filesystem vs Pootle
- FileMapper - maps the file layout on the filesystem to the expected Pootle layout
- Store de/serialization - makes it possible to customise and adapt file serialisation, most likely for slight deviations from the official file format specification.
- Removed the ability to add new TPs from the admin UI for Pootle FS projects, we will initialise new TPs differently in Pootle FS.

### Development changes

- Updated and pinned PyPI requirements:
  - From now on requirements will be pinned in order to simplify support and development.
- Tests:
  - Massive improvement in test framework.
  - Coverage increased from 55% to 94%.
  - Moved to tox.
  - Travis caching and optimisations.
  - Added JavaScript testing.
- Code sanity:
  - Python code cleanup/linting pep8/pyflakes/pep257 to increase code health.
  - Javascript code linting and cleanups.
  - CSS code linting and cleanups.
- Code polishing:
  - Moved all commands to argparse.
  - Moved shortcuts to Mousetrap.
  - JS improvements, move to React components.
- Triage meetings are now held on a weekly basis.

### Command changes and additions

- Running Pootle commands using manage.py is no longer supported, use pootle instead.
- pootle command warns if configuration is missing.
- Changed commands:
  - `verify_user` and `purge_user` now accept multiple usernames.
  - `refresh_scores` now recalculates user scores and accepts multiple usernames. It can be run across projects and/or languages.
  - `contributors` command has been refactored in order to return more accurate results and has new options `--since`, `--until` and `--mailmerge`. The `--from-revision` option has been removed.
  - `flush_cache` flushes `default`, `redis` caches, accepts `--rqdata`, `--django-cache` options.
  - `export` is now able to export zipped TMX files per translation project with the `--tmx` option. `--rotate` option allows old files to be removed.
  - `init` now creates a development configuration with `--dev` option.
- Added new commands:
  - `list_serializers` allows to view serializers and deserializers installed on your system.
  - `config` allows to get, set, list, append and clear configuration settings.
  - `init_fs_project`.
  - `set_filetype`.
  - `schema` allows to dump the database schema on MySQL which is useful for diagnosing differences in database schema.
  - `update_data` allows to update the stats data.
- Removed commands:
  - run_cherrypy.
  - start has been removed, use `runserver` instead.
  - refresh_stats.
  - clear_stats.

### Changes in settings

- Changes in settings:
  - MySQL database connections should now use `STRICT_TRANS_TABLES`.
  - `POOTLE_TM_SERVER` no longer receives the `MIN_SCORE` parameter, as it was misleading and had questionable effects.
  - `POOTLE_TM_SERVER` now accepts a `MIN_SIMILARITY` parameter, to filter out results which might be irrelevant. To learn more, check the documentation on `MIN_SIMILARITY`.
  - Changed the default value for ACCOUNT_SESSION_REMEMBER so now sessions are always remembered.
  - `POOTLE_MARKUP_FILTER` defaults to `'markdown'`, and `None`, `'html'`, `'textile'` and `'restructuredtext'` values have been deprecated. Deployments using any deprecated markup must migrate manually to Markdown. This setting will be removed in the future since Markdown will be the only available markup.
- Added new settings:
  - `POOTLE_SCORES` accepts custom settings for user scores calculation.
  - `POOTLE_SEARCH_BACKEND` to allow configuring the search backend to be used.
  - `POOTLE_EMAIL_FEEDBACK_ENABLED` to allow disabling sending emails to suggesters when suggestions are accepted or rejected.
  - `POOTLE_CUSTOM_LOGO`, `POOTLE_FAVICONS_PATH`, `POOTLE_FS_WORKING_PATH` and `POOTLE_CANONICAL_URL` settings to allow easy customisations.
  - `POOTLE_SQL_MIGRATIONS`.
  - `AMAGAMA_SOURCE_LANGUAGES`.
- Removed settings:
  - `POOTLE_QUALITY_CHECKER` since the custom quality checkers feature is gone.
  - `POOTLE_SCORE_COEFFICENTS` has been removed and replaced with `POOTLE_SCORES`.

## Credits

This release was made possible by the following people:

Ryan Northey, Dwayne Bailey, Julen Ruiz Aizpuru, Taras Semenenko, Leandro Regueiro, Igor Afanasyev, Claude Paroz, Safa Alfulaij, Rene Ladan, Kevin Scannell, Jason P. Pickering, Eamonn Lawlor, Alexander Lakhin, Robbie Cole, Rhoslyn Prys, Prasasto Adi, Nootan Ghimire, Mikhail Paulyshka, Mike Robinson, leonardcj, Henrik Feldt, Francesc Ortiz, Allan Nordhøy, Christian Lohmaier, Burhan Khalid, benbankes, Arash Mousavi, Andy Kittner, Adam Chainz.

And to all our bug finders, testers and translators, a Very BIG Thank You.