A config-driven pipeline for generating and filtering fine-tuning data with multi-source ingestion, Evol-Instruct, MinHash and LSH deduplication, and SQLite checkpoints.
GitHubView repository →
1 project
A config-driven pipeline for generating and filtering fine-tuning data with multi-source ingestion, Evol-Instruct, MinHash and LSH deduplication, and SQLite checkpoints.