Skip to content

GitLab

J
jingxiaoguan
Open
Opened by weidongying@weidongyingMaintainer

feat: 合同 ZIP 批量异步解析与持久化进度

Problem Statement

合同导入当前在浏览器请求中同步执行 MinerU、字段抽取与草稿落库。大文件、ZIP、多合同场景会长时间阻塞并超时；刷新页面后用户无法获知进度，也无法可靠恢复失败合同。多个选中的文件还会被错误视为同一合同。

Solution

将合同导入改为持久化的异步解析任务。上传阶段只校验、保存原件并按合同边界创建任务；全部文件落盘后，后台按队列顺序执行 MinerU、关键词识别和字段抽取。任务、进度、失败原因和草稿关联均持久化，可在页面刷新或服务重启后恢复。

User Stories

  1. As a 合同管理员, I want to upload a ZIP up to 1GB, so that I can import a batch of contracts in one operation.
  2. As a 合同管理员, I want each ZIP top-level folder to become one contract package, so that its attachments remain associated with one contract.
  3. As a 合同管理员, I want each ZIP top-level loose file to become an independent contract, so that unrelated files are never merged.
  4. As a 合同管理员, I want each individually uploaded file to become an independent contract, so that batch selection does not incorrectly merge contracts.
  5. As a 合同管理员, I want upload completion to be separate from parsing, so that large files do not hold the browser request open.
  6. As a 合同管理员, I want queued contracts to parse sequentially, so that model and MinerU capacity is controlled.
  7. As a 合同管理员, I want a right-top progress panel, so that I can see the current file, overall progress and result.
  8. As a 合同管理员, I want progress to survive refresh and reopening the page, so that I do not lose visibility of background work.
  9. As a 合同管理员, I want parsing contracts to appear in the ledger as 解析中, so that I know the upload was accepted.
  10. As a 合同管理员, I want failed contracts to appear as 解析失败 with the error reason, so that I can take action.
  11. As a 合同管理员, I want parsing and failed records to be view-only, so that incomplete extraction cannot be edited or confirmed.
  12. As a 合同管理员, I want to retry a failed contract using DeepSeek, so that a local-model or JSON failure has a deterministic fallback.
  13. As a 合同管理员, I want successful parsing to become 待核对, so that the existing manual review and vectorization flow remains unchanged.
  14. As a 系统管理员, I want task lifecycle operations recorded, so that uploads, retries and failures are auditable.
  15. As a 查询用户, I want queued, parsing and failed contracts excluded from formal SQL and RAG results, so that only reviewed formal contracts are searchable.

Implementation Decisions

  • Add a persistent contract parse job and extend contract package status to cover queued, running, draft, confirmed and failed. Store package identity, source files, queue order, current file, progress, attempt count, selected extraction provider, error text, draft reference and timestamps.
  • Keep contract package/source-file records as the attachment truth. ZIP extraction is safe: reject path traversal, preserve a top-level folder as one package, and map each top-level loose supported file to its own package. Nested paths remain inside their top-level package.
  • Upload endpoint only persists files and creates jobs. ZIP maximum is 1GB; individual supported attachment validation remains enforced. Browser upload progress and persistent parse progress are separate states.
  • A Parse Service background worker claims one queued job at a time from PostgreSQL, records running/progress state, creates a draft on success, and records a recoverable failure on exception. Startup resumes queued/running jobs without depending on browser state.
  • Initial extraction uses the existing Qwen-first strategy. Explicit retry of a failed job uses DeepSeek only. Both respect the existing 50-page parsing, keyword and vectorization limit.
  • Expose authenticated Gateway APIs to create jobs, list active/recent jobs, read a job/its read-only preview and retry failed jobs. The front end never talks to PostgreSQL or Parse Service directly.
  • Extend ledger response/view state with parsing and failed review states. These records expose a read-only view; edit, keyword override, confirm and vectorization controls are disabled until a draft succeeds.
  • Add a global progress panel in the upper right using the current system theme, restored from task API on application load and polling while active jobs exist.
  • Existing confirmed contracts and their vectors are untouched. Existing manual review→confirmed→vectorization behavior remains the only route into the formal contract library.

Testing Decisions

  • Primary seam: API + PostgreSQL integration tests covering upload enqueue, persisted state after a new app instance, sequential claim, success-to-draft, failure persistence and DeepSeek retry. Assert externally observable status, progress, draft reference and source association rather than worker internals.
  • Add ZIP classification tests: top-level folders become packages, loose top-level files become packages, nested files stay with their parent, unsafe archive paths are rejected and 1GB policy is validated without allocating 1GB.
  • Add Gateway multipart policy tests for the ZIP and aggregate limits plus route proxy behavior.
  • Add Vue tests or component-level behavioral checks for the persistent progress panel and read-only restrictions; production build remains required.
  • Reuse existing parse-service ingestion, source persistence, draft confirmation and keyword-rescan test patterns where applicable.

Out of Scope

  • Browser direct folder upload.
  • Full-document processing after page 50.
  • Automatic contract-order association or mixed business statistics.
  • Changing the 59 confirmed contracts, their manual review state or existing vectors.
  • Automatic retry loops after failure; retry remains a deliberate DeepSeek action.

Further Notes

The system currently has a distinction between formal contract confirmation and page review status. The new asynchronous statuses describe pre-draft parse lifecycle only; they must not be interpreted as formal, searchable contracts. Progress must remain available after a server restart by reusing persisted jobs.