[16.0][ADD] dms_import: Migration data from documents EE to dms CE #409

xaviedoanhduy · 2025-04-17T10:51:38Z

Purpose

This migration script allows moving data from Enterprise documents* (EE) to the OCA dms* (CE) modules.

The goal is to preserve:

Folder hierarchy
Access rights (read/write groups)
Tags and tag categories
Files and their attachments

Approach

The migration is implemented as a post-init hook and works directly at the SQL level to avoid dependencies on EE models. This ensures:

Compatibility with CE environments (no EE models required).
Preservation of hierarchy, security, and data.

The migration flow has three steps:

1. Tags migration

documents.facet → dms.category
documents.tag → dms.tag
Prevents duplicates by checking existing categories and tags.

2. Folders migration

documents.folder → dms.directory
Maintains parent/child relationships.
Root folders receive a default dms.storage.
Access rights migrated via:
- documents_folder_res_groups_rel → dms.access.group (Write groups)
- documents_folder_read_groups_rel → dms.access.group (Read groups)
Folder-level tags are also migrated via facet → tag mapping.

3. Files migration

documents.document (binary only) → dms.file
Preserves folder assignment and file metadata.
Keeps linked attachments by updating ir.attachment.res_model/res_id to the new dms.file.
Migrates file tags using the tag mapping.
Uses batch processing (1000 docs per batch) for scalability.

Data Mapping Details

EE Model / Table	DMS Model / Table	Notes
`documents.facet`	`dms.category`	Folder tag categories → Categories
`documents.tag`	`dms.tag`	Tag names + facet → Tags, with deduplication
`documents.folder`	`dms.directory`	Folder hierarchy → Directory hierarchy, sequence → color (fallback)
`documents_folder_res_groups_rel`	`dms.access.group`	Write group permissions (create/write/unlink)
`documents_folder_read_groups_rel`	`dms.access.group`	Read-only group permissions
`documents.folder.facet_ids.tag_ids`	`dms.directory.tag_ids`	Folder-level tags migrated via mapping
`documents.document (binary)`	`dms.file`	Binary documents only, skipped inactive or non-binary docs
`documents_document_tag_rel`	`dms_file_tag_rel`	Many2many doc ↔ tags mapping
`ir.attachment` (linked to document)	`ir.attachment` (relinked)	Updated to point to new `dms.file`

Extra Notes

Colors default to randint(1, 11) when no valid sequence color is available.
A default database storage (dms.storage) is created if none exists.
Logging is verbose:
- Created directories
- Assigned groups
- Migrated tags and files
Errors are logged per record without blocking the migration.

Example Workflow

Run module installation with the migration hook.
Check that:
- Categories and tags exist in dms.category and dms.tag.
- Directories are created with correct hierarchy.
- Access groups are properly assigned.
- Files are migrated and linked to attachments.
- Tags are assigned to both directories and files.

Validation

Before migration: Ensure EE tables (documents_*) exist and contain data.
After migration: Validate:
- Number of categories, tags, and directories.
- Randomly sample files to confirm attachments and tags.
- Check group permissions are correctly migrated.

Migration should complete successfully and log a summary:

Successfully migrated X tags.
Successfully migrated Y folders (Z new root directories).
Successfully migrated N files.
Migration completed successfully.

kobros-tech · 2025-05-29T10:55:09Z

@xaviedoanhduy

could you write desctiption on how to use it or configure if necessary, so that we could review easily?

nilshamerlinck · 2025-05-29T11:01:46Z

hi @kobros-tech you will receive an invitation to join the slack workspace where you'll find more information, see you there ;-)

kobros-tech · 2025-05-29T12:39:28Z

@nilshamerlinck
thanks, I have now a stack account!

kobros-tech

Implement Review Only, would recommend adding test cases

kobros-tech · 2025-06-01T13:11:57Z

@xaviedoanhduy

what can happen if there is a file that should be for a specific partner, shall we get the same file accessible by the same partner Only after migration?

xaviedoanhduy · 2025-06-01T16:38:17Z

@kobros-tech,

As far as I recall, dms.file/dms.directory are shared at the user - res.users level, while documents.document/documents.folder are assigned to contacts - res.partner.

I’d appreciate your thoughts if you have any ideas on how to map these fields between the two models.

kobros-tech · 2025-06-01T16:58:12Z

all right I will ping other mind to tell about real life scenario and then we can apply

@wlin-kencove

xaviedoanhduy · 2025-07-30T07:22:17Z

@kobros-tech,

As far as I recall, dms.file/dms.directory are shared at the user - res.users level, while documents.document/documents.folder are assigned to contacts - res.partner.

I’d appreciate your thoughts if you have any ideas on how to map these fields between the two models.

Sorry for my omission, I'd like to clarify some points:

For your question, it seems that the answer is no. because the data from EE documents is not shared with any user (it can only be owned by 1 partner) but only through links or permission groups - I will give an example that the Internal folder has Read Groups (Write Groups) containing the Documents/Adminstrator permission group and the user named Demo is not in this group, he will not be able to see the above folder in the backend view - but he can have full read and write permissions if he owns the link generated from Share links.
EE documents, the documents will mostly not be accessible flexibly to portal users (even if that user is the owner of that document - it will only be seen through the backend in the Contact app). And for portal users (or even non-logged in users), they only need to know the 1 link created from Share links (documents.share model) - and that is the only way they can access these documents.
For CE dms, these sensitive records are controlled from another group model (dms.access.group) and allow all public users to see these documents if their group (res.groups - portal group) or maybe on users (res.users).
For the context of the current module, I am using the mechanism of creating dms.access.groups based on the name of the document and Read Groups (Write Groups) -> resulting in only users in the above permission groups can access it - and one thing is for sure, those permission groups do not contain portal users.

kobros-tech · 2025-07-30T11:01:14Z

yes, it is much better.

once you are done you can ask me to review, good luck!

[IMP] dms_import: move pre_init_hook to post_init_hook [REF][FIX] dms_import: Use new syntax, avoid sql injection and fix group creation bug dms_import: also migrate achived data

…default group permissions

andreampiovesana · 2025-10-16T12:07:42Z

nice to have

…heavy compute fields

kobros-tech approved these changes Jun 1, 2025

View reviewed changes

xaviedoanhduy force-pushed the 16.0-add-dms_import branch from d91427c to 1650324 Compare July 31, 2025 07:45

xaviedoanhduy changed the title ~~[16.0][ADD] dms_import~~ [16.0][ADD] dms_import: Migration data from documents EE to dms CE Sep 25, 2025

xaviedoanhduy force-pushed the 16.0-add-dms_import branch 2 times, most recently from 1ebe9f8 to de8ccd8 Compare September 29, 2025 09:42

[ADD] dms_import: Migration data from documents EE to dms CE

d98f346

[IMP] dms_import: move pre_init_hook to post_init_hook [REF][FIX] dms_import: Use new syntax, avoid sql injection and fix group creation bug dms_import: also migrate achived data

xaviedoanhduy force-pushed the 16.0-add-dms_import branch from de8ccd8 to d98f346 Compare September 29, 2025 10:19

[REF] dms_import: avoid duplication by forcing tags, categories, and …

f755c12

…default group permissions

xaviedoanhduy force-pushed the 16.0-add-dms_import branch 2 times, most recently from 9844447 to e06570c Compare October 16, 2025 09:36

xaviedoanhduy force-pushed the 16.0-add-dms_import branch 2 times, most recently from 90fe58e to 37ee298 Compare October 16, 2025 12:24

[IMP] dms_import: improve performance, reduce batch size, and bypass …

951cea0

…heavy compute fields

xaviedoanhduy force-pushed the 16.0-add-dms_import branch from 37ee298 to 951cea0 Compare October 16, 2025 12:32

xaviedoanhduy added 2 commits October 17, 2025 15:07

[FIX] dms_import: handle duplicate name

e458823

[FIX] dms_import: standardize file names

36420ab

xaviedoanhduy force-pushed the 16.0-add-dms_import branch from c1abd37 to 36420ab Compare October 17, 2025 10:58

[IMP] dms_import: improve unique_name_new to avoid long name

c98ecf0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[16.0][ADD] dms_import: Migration data from documents EE to dms CE #409

[16.0][ADD] dms_import: Migration data from documents EE to dms CE #409

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

[16.0][ADD] dms_import: Migration data from documents EE to dms CE #409

Are you sure you want to change the base?

[16.0][ADD] dms_import: Migration data from documents EE to dms CE #409

Uh oh!

Conversation

Uh oh!

Purpose

Approach

1. Tags migration

2. Folders migration

3. Files migration

Data Mapping Details

Extra Notes

Example Workflow

Validation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants