Training Auto-Classification Models for RIM Bot

To use the RIM Bot to auto-classify documents in the Document Inbox, a Trained Model must be trained and deployed. This training allows the machine learning model to learn from your inputs, preparing it to intelligently process data.

Vault automatically creates a Trained Model record of the Document Classification type in all RIM Vaults with at least 1,500 Steady state documents. As long as a Trained Model is not already deployed, Vault also deploys the model for you. In every Vault, custom and system-trained models refresh with each Vault release.

This process occurs once per release, so if you wish to update your Trained Model at any time (for example, to reflect new document types, or to attempt to improve your results), you must follow the process described here to train, evaluate, and deploy it.

How Auto-trained Models Work

The RIM Bot is auto-on for all RIM Submissions customers with more than 1,500 Steady state documents. The process the system uses to create, train and deploy a Document Classification Trained Model is as follows:

1. The Auto-Train Models job runs each night at 1:00am EST on all production and pre-release¹ Vaults. This job will check that:
   • No system-created model has been created since the last major release
   • There are at least 1,500 Steady state documents in this Vault
2. The job creates a Trained Model of the Document Classification type with the below default values. If your previously-deployed model included Excluded Classification records or criteria in the Document Criteria - VQL field, Vault copies them to the new trained model.
   • Prediction Confidence Threshold: 0.85
   • Minimum Documents per Document Type: 10
   • Auto-Deploy: Yes (true)
3. The latest document versions (based on the Version Created Date) that fall into the following categories are used to train this model:
   • In a Steady state (Approved/Final)
   • Not a Binder
   • Not in an unsupported document type
   • The document has pages
4. If there is already a deployed model of the same Trained Model Type in this Vault, the auto-trained model stays in the training state, otherwise it will be automatically deployed after it finishes training.
5. Once the auto-trained model is deployed, any documents uploaded to the Inbox may be auto-classified by the RIM Bot.

Auto-classification models are automatically refreshed with each Vault general release. If you are currently using a manually-trained model, Vault makes a copy of that model, trains it, and deploys it to replace the old model. Otherwise, if you are currently using the previous release’s auto-trained model, Vault deploys this new model to replace the old Trained Model. This ensures the system is training on the latest documents that represent your document hierarchy.

Given the number of RIM customers, auto-training and deploying models in your Vault may take 48 to 72 hours after each general release.

How To Train a Model

Like all machine learning tools, the RIM Bot requires input to learn before performing tasks on its own. Generally, the larger and more accurate the inputs, the better the resulting model will be. Vault stores accumulated input in Trained Model object records.

Prediction Confidence

Vault uses a Prediction Confidence score to indicate how certain RIM Bot is that its prediction is correct. This value is between 0 (likely wrong) and 1 (likely correct). The better your inputs, the higher the Prediction Confidence will be. Vault stores Prediction Confidence scores in Prediction object records.

Prediction Confidence Threshold

Vault uses the Prediction Confidence Threshold field value on a Trained Model record to determine what score is required before the model can use that Prediction. The Prediction Confidence Threshold is system-managed and set to 0.85 by default. This means that if the Prediction Confidence for a document uploaded to the Document Inbox is .8728, Vault auto-classifies the document.

Creating a Document Classification Trained Model

Before creating a Trained Model, carefully consider the following limitations:

• Vault allows Admins to train models in Pre-release or Sandbox environments using their production environment documents, verifying the training process. These models, however, cannot be moved to your production Vault, so Trained Models must be created and trained in the production environment as well.
• Certain categories of documents cannot be auto-classified or used in model training. These include:
  • Audio and video files
  • Non-text files, such as ZIP files, statistical files, or database files
  • Documents where Vault cannot extract text, for example, if the text is too blurry.
• We recommend using at least 3,000 documents in steady states, such as Approved or Final, to train the machine learning model. You may use RIM Bot on Vaults with 1,000-3,000 documents, however, this may limit the quality of your predictions.
• If any inputs are misclassified documents, predictions may be negatively impacted. For example, if several documents that should have been classified as Regulatory > Correspondence > Approval Letter were classified as Regulatory > Correspondence > Agency Decisions, RIM Bot will be less confident about predictions for those document types.

Creating the Trained Model Object Record

1. Navigate to Admin > Configuration > Document Fields and review your Vault’s configuration for the RIM Auto Classification and Tags fields. In order for users to observe the auto-classification process in their Document Inbox:
   • The Unclassified document type must use the RIM Auto Classification field.
   • Field-level security for the Tags field must be configured as Read Only or Editable.
2. Navigate to Admin > Business Admin and click into the Trained Model object.
3. Click Create.
4. For the Trained Model Type, select Document Classification.
5. Click Save.

After creating the Trained Model object record, optionally add any Excluded Classifications, then train the model.

Creating Excluded Classifications

You can define classifications that will be excluded from your Trained Model. The RIM Bot excludes the specified classification(s) from all extraction, training, and testing during model deployment. Additionally, later predictions the RIM Bot makes are not actioned if a document is in (or predicted to be in) an excluded classification.

You can specify excluded classifications before or after a model is trained. If you add an excluded classification after the model’s training, the model is not automatically retrained. However, the RIM Bot does not take any action against documents of the excluded classification.

This exclusion applies only to the Trained Model to which the Excluded Classification belongs. If you create an Excluded Classification for a Trained Model which is no longer in use, you must re-define it for the currently-deployed model.

To create an excluded classification:

1. Under Excluded Classifications, click Create.
2. Select the Status of the Excluded Classification.
3. Select the Classification you wish to exclude.
4. Enter any relevant Comments.
5. Click Save.

Training the Trained Model

Once you have created the Trained Model, perform the Train Model action and click Start. The Trained Model record moves to the In Training state.

To train your model, Vault sets the Training Window Start Date and pulls all non-Archived documents in a Steady State, such as Approved or Final, with a Version Created Date value between the Training Window Start Date and the current date. If there are more than 200,000 documents that fit this criteria, Vault uses the 200,000 most recent documents.

Additionally, an asynchronous job tracks two activities as part of training:

1. Document Extraction: During this process, the system collects the data from the document set. The output is a CSV file (document_extract_results.csv) in which an Admin can see which documents were able to be used as input and which were not attached under Trained Model Artifacts. Vault sends a notification to the Admin who started the action when the extraction is complete.
2. Model Training: During this process, the system will use 80% of the extracted data to build a machine learning neural network model, then test that model using the remaining 20%. The output is a number of performance metrics in both the Trained Model Performance Metrics object and attached CSVs under Trained Model Artifacts. Vault sends a notification to the Admin who started the action when training is complete.

The time required to complete these jobs varies depending on the number of documents used as input: About 1 hour for Vaults training on 3,000 documents, to about 24 hours for Vaults training on 200,000 documents.

Once model training is complete, the Trained Model record moves to the Trained state.

Training a Trained Model in Pre-Release or Sandbox Environments with Production Data

You can train a Trained Model in your Pre-Release or Sandbox Vault with production documents for evaluation purposes. You cannot move the resulting Trained Model to your production environment.

To train using production data, run the Train Model From Production Data action. This action is only visible in Pre-Release and Sandbox Vaults.

After evaluating your Trained Model, you’ll need to perform training again in your production Vault to begin using RIM Bot features there.

Evaluating the Trained Model

Vault provides key metrics you can reference in the Trained Model record’s Training Summary Results field to evaluate your model: Extraction Coverage, Auto-classification Coverage, and Auto-classification Error Rate. See the definitions for these metrics and how to improve them.

Deploying the Trained Model

Once you have trained and evaluated your Trained Model, select the Deploy Model action from the Trained Model record, review the prompt to ensure you agree with the outcome and click Start. The Trained Model record will move to the In Deployment state.

An asynchronous job tracks the deployment of this Trained Model in your Vault. The time required to complete these jobs varies, and it can take anywhere from 30 minutes to two (2) hours. Vault sends a notification to the Admin who performed the action when deployment is complete.

Once the deployment job finishes, the Trained Model record moves to the Deployed state and Vault begins auto-classifying the documents in the Document Inbox.

Only one (1) Trained Model per Trained Model Type can be deployed at a time.

Replacing a Deployed Trained Model

To replace a deployed model with a new Trained Model, simply deploy the new model. It replaces the currently active model, and auto-classification is not interrupted. This is the recommended method for replacing models.

Refreshing a Deployed Trained Model

To refresh a deployed model, select the Refresh Model action. It will automatically create a deep copy of the current Trained Model and start the training process. This action prevents users from starting multiple training jobs simultaneously, and refreshes a Trained Model in fewer steps.

Additional Trained Model Actions & Details

You can only have five Trained Models per Trained Model Type. If you attempt to train a sixth, Vault advises you to archive a model before training another. To do so, select the Archive Model action on a Trained Model record. The Trained Model record moves to the Archived state. Archived models are not recoverable.

You can also remove deployed models and disable auto-classification by using the Withdraw Model action on a Trained Model in the Deployed state. Doing so moves the Trained Model record back to the Trained state.

About the Prediction Object

When a Trained Model is deployed and used to predict data for a document, the Prediction object keeps track of each individual prediction attempt. It’s unlikely that Admins will need to work with this object directly, but it may be useful to understand the object fields:

• Prediction ID: Unique identifier for that prediction, automatically assigned by Vault
• Related Record Unique ID: Identifier for the file being evaluated, automatically assigned by Vault
• Related Record: Metadata for the document being evaluated, formatted as JSON. You can locate the Vault Document ID, Major version, and Minor version here if needed.
• Predictions: The prediction data for this attempt from RIM Bot, formatted as JSON. You can use this field to understand if a prediction failed and why; which Trained Model was used to make the prediction; and, in the case of Document Classification, the first, second, and third top predictions from the model along with their Prediction Confidence scores. If the first Prediction score is above the deployed Trained Model Prediction Confidence Threshold, the document will have been auto-populated with that prediction. This can also be seen with the auto-populated JSON parameter.
• Feedback: Post-prediction activity. This field shows the current value for the data being predicted in the trueValue JSON parameter and if that value matches the corresponding first Prediction in the Predictions field in the trueValueMatch JSON parameter.
• Additional Details: Lists from where Vault generates the prediction. This can include multiple sources.

About the Prediction Metrics Object

When a Trained Model is deployed and used to predict data for a document, the Prediction Metrics object keeps track of the model’s performance over time. The Prediction Metrics job runs monthly and generates records that track the overall Trained Model performance, as well as performance per document classification.

You can view the following object fields from the Trained Model page layout:

• Model Performance ID: Unique ID, assigned by Vault
• Created Date: Date the prediction metric was calculated
• Trained Model Type: The Trained Model Type being evaluated, for example Auto-Classification
• Metric Type: Metric type presented
• Metric Subtype: Subtype of the metric presented
• Number of Documents: The number of documents sent to the RIM Bot during the given time period.
• Documents Extracted: The number of documents sent to the RIM Bot that had text successfully extracted and evaluated by the model.
• Extraction Rate: The rate at which documents sent to the RIM Bot had their text successfully extracted (Documents Extracted divided by Number of Documents).
• Documents with Predictions: The number of documents with a predicted value. For Auto-Classification, these are all documents sent to the RIM Bot.
• Correct Predictions: The number of times the predicted value was accurate, whether or not the RIM Bot acted upon it. For Auto-Classification, the predicted classification is correct whether or not it is above the Prediction Confidence Threshold.
• Predictions Above Threshold: The number of times the RIM Bot acted upon the prediction. For Auto-Classification, this means the predicted classification was above the model’s Prediction Confidence Threshold.
• Correct Predictions Above Threshold: The number of times the predicted value was accurate and the RIM Bot acted upon it. For Auto-Classification, this means the RIM Bot set the correct classification on the model.
• Success Rate: The rate at which predictions on which the system acted were confirmed as true predictions (Correct Predictions Above Threshold divided by Predictions Above Threshold)