Learn from Curated Repository Collection

[alirezamshi]

Description

Learn from ~2k curated repositories in ML, data, and quant domains. Extract knowledge and serve via skills.leeroo.com.

Data Source

Repository list: https://api.leeroo.com/storage/v1/object/public/avatars/f47ac10b-58cc-4372-a567-0e02b2c3d479/file_list.json

Domains covered:

• Machine Learning
• Data Engineering
• Quantitative Finance

Goal

1. Run repo learner (issue Implement Repository Learner #7 ) on all repos in the list
2. Extract patterns, abstractions, and best practices
3. Index into knowledge graph
4. Serve at https://skills.leeroo.com

Considerations

• Batch processing for 2k repos
• Rate limiting for GitHub API
• Incremental updates (new repos, repo changes)
• Quality filtering (stars, activity, code quality)

References

• related to repo learner implementation Implement Repository Learner #7
• src/knowledge/learners/ - Learner implementations

[alirezamshi]

the process is implemented here #40

[alirezamshi]

samples of ingested repos are in "data/wikis" in this commit

[alirezamshi]

rough cost calculation of bedrock for using opus 4.5 for 2k repos is around 20K$.