Understanding Concept Clustering in QuikData

Overview

Concept Clustering is an automated document analysis feature that discovers the main themes and topics within your document collection. Rather than requiring you to manually read and categorize thousands of documents, Concept Clustering uses machine learning to identify natural groupings based on the actual content of your documents.

This feature is particularly valuable in eDiscovery and document review scenarios where you need to quickly understand what a large collection of documents is about, prioritize review, or find documents related to specific themes.

How It Works

Step 1: Text Extraction and Preparation

Before analysis can begin, the system extracts the text content from each document in your folder. During this phase:

• Email headers are removed — Metadata like "Priority: Normal" and routing information is stripped out since it doesn't represent the document's actual content.
• Common boilerplate is filtered — Legal disclaimers, email signatures, and standard phrases like "Please see attached" are removed to focus on meaningful content.
• Very short documents are excluded — Documents with insufficient text (less than ~20 words of meaningful content) cannot be reliably categorized and are assigned to a special "Contains Insufficient Text" concept.

Step 2: Building the Topic Model

The system uses an algorithm called Latent Dirichlet Allocation (LDA) to discover topics. Here's the intuition behind how it works:

Imagine you have a collection of news articles. Some are about sports, some about politics, some about technology. LDA works backward from the words in each document to figure out what topics exist and which documents belong to which topics.

The algorithm assumes:

• Each topic is characterized by certain words appearing together frequently (e.g., a "sports" topic might feature words like "game," "score," "team," "championship")
• Each document can contain multiple topics in different proportions (e.g., an article about a stadium financing deal might be 60% sports, 40% business)

The number of concepts discovered scales automatically based on your collection size:

Step 3: Identifying Distinctive Vocabulary

Once topics are discovered, the system identifies the most distinctive words for each concept. This is where c-TF-IDF weighting comes in.

Not all words are equally useful for understanding what makes a topic unique:

• Words like "company," "agreement," or "meeting" might appear in almost every business document
• Words like "merger," "acquisition," or "valuation" are more distinctive and informative

The c-TF-IDF algorithm boosts words that are distinctive to specific topics while dampening words that appear across all topics. This ensures each concept is labeled with the words that best represent what makes it unique.

Example:

• If "Walter White" appears primarily in Topic 3 documents, it gets boosted as a distinctive term for that concept
• If "meeting" appears equally across all topics, it gets dampened and likely filtered out

Step 4: Classifying Documents

Finally, every document in your folder is analyzed against the discovered topics. Each document receives:

• Topic assignments — A document can belong to multiple concepts if it covers multiple themes
• Confidence weights — Each assignment includes a weight (0-100%) indicating how strongly the document relates to that concept

Documents are assigned to a concept when the confidence weight reaches 20% or higher. This threshold ensures meaningful assignments while allowing documents to belong to multiple relevant concepts.

What You'll See

After Concept Clustering completes, each concept is displayed with:

1. Concept ID — A numeric identifier (0, 1, 2, etc.)
2. Key Terms — The most distinctive words and phrases for that concept, ranked by relevance
3. Document Count — How many documents are associated with this concept
4. Weight Distribution — How strongly documents are associated with this concept

Example Output:

Technical Details

For those interested in the underlying methodology:

Algorithm: Latent Dirichlet Allocation (LDA)

LDA is a generative probabilistic model that treats documents as mixtures of topics, where each topic is a probability distribution over words. The implementation uses:

• 200 iterations for model convergence
• Bigram support — Both single words and two-word phrases are considered (e.g., "intellectual property" as a single term)
• Vocabulary scaling — The maximum vocabulary size scales with collection size to balance precision and performance

Vocabulary Weighting: c-TF-IDF

Class-based TF-IDF (c-TF-IDF) is applied as a post-processing step to improve concept labeling:

score = TF × log(1 + N / n_t)

Where:
- TF = term frequency in the topic
- N = total number of topics  
- n_t = number of topics containing the term

Terms appearing in only one topic receive ~4x boost; terms appearing in all topics are dampened to ~0.7x their original score.

Performance Characteristics

• Sampling — For collections over 2,500 documents, a representative sample is used for model training (all documents are still classified)
• Parallel processing — Document classification runs in parallel batches of 500 documents
• Memory efficiency — Large documents are truncated to 500KB of text to prevent memory issues

Best Practices

For best results:

1. Ensure documents have extractable text — Scanned images without OCR will appear as "Insufficient Text"
2. Larger collections produce better results — LDA works best with hundreds or thousands of documents
3. Review the key terms — The automatically generated labels give you a quick understanding of what each concept represents
4. Use concepts for prioritization — Focus your review on the concepts most relevant to your case

Limitations to be aware of:

• Very short documents (emails with just "Thanks!" or "See attached") cannot be meaningfully categorized
• The algorithm discovers statistical patterns—concept labels are generated from word frequencies, not semantic understanding
• Documents covering multiple topics will appear in multiple concepts, which is expected behavior

Frequently Asked Questions

Q: Why is a document in multiple concepts?
A: This is normal and expected. A document discussing both "merger negotiations" and "employee retention" would naturally belong to concepts covering each of those themes.

Q: What does "Contains Insufficient Text" mean?
A: These documents didn't have enough meaningful text content after filtering to be reliably categorized. This commonly includes very short emails, documents that are primarily images, or files that couldn't be processed.

Q: Can I customize the number of concepts?
A: The system automatically determines the optimal number of concepts based on your collection size. This ensures meaningful groupings without over-fragmenting smaller collections or under-representing larger ones.

Q: How long does Concept Clustering take?
A: Processing time depends on collection size. A folder with 500 documents typically completes in under 30 seconds. Larger collections of 50,000+ documents may take several minutes.

Q: Are the results deterministic?
A: Yes. Running Concept Clustering on the same folder will produce the same results, allowing for reproducibility in legal and compliance contexts.