Software engineering datasets fall into two groups. Public ones, built from open-source repositories and research projects, are free to study but limited in license and scope. Private ones, such as your team’s Slack history, tickets, codebase and CRM records, are the data AI buyers have the hardest time finding, and they can be licensed. If you run a company with years of engineering and customer records, the useful question is what yours is worth to the buyers who want it. We run a sell-side brokerage that asks multiple AI data buyers for sealed bids on de-identified company data, so we look at this market from the seller’s chair.
This guide covers what public datasets contain, how they are accessed, what their licenses allow, and what that tells you about preparing your own data. It also covers website behavior data, which turns up in the same buyer conversations.
TL;DR
- Public software engineering datasets cluster around version control, issues, defects and code tasks. Project management and real internal team activity are barely represented.
- Most public sets come with license terms. Some are free only for non-commercial use, so they are poor models for a commercial deal.
- Website behavior datasets exist in public form (clickstream, sessions, page loads), but they are usually hashed, aggregated or short-lived samples.
- Companies with 3+ years of internal engineering, support and behavior records hold the kind of data public sets lack.
- We invite multiple qualified buyers to bid on your data, charge sellers nothing, and move no data until a deal is signed.
Table of Contents
- What software engineering datasets exist today
- The gap in public software engineering data
- Website behavior data: what public sets capture
- How people access public datasets
- Licenses and privacy: why public terms are a poor template
- What to evaluate before you prepare your data
- Why one offer is rarely enough
- How the process works with us
- Where to start
- FAQ
- Sources
What software engineering datasets exist today
A 2025 academic directory of mining-software-repository datasets sorts them into six areas: version control, software issues, developer metrics, software evolution, semantic metrics, and other data such as Dockerfiles, Jupyter notebooks and license texts. Examples include the Public Git Archive for version control, Apache Jira issue collections, and a dataset covering 44 years of Unix evolution.
A few representative ones show the range:
- Issue and project data. The TAWOS dataset holds 508,963 issues from 44 open-source Agile projects, with change logs, comments, sprints, versions and story points.
- Defect data. A 2025 survey catalogs 151 software defect datasets, and over 60% of them focus on general software development.
- AI task data. SWE-rebench-V2 packages 32,079 software engineering tasks taken from real GitHub issues and pull requests across 20 languages. Each record carries the repository, base commit, reference patch, test patch, problem statement and license.
- Research repositories. The tera-PROMISE repository offers free, long-term storage for software engineering research data.
The pattern is clear. These collections come from public code hosting and public trackers, because that is what researchers can legally and cheaply gather.
The gap in public software engineering data
A 2024 classification of open-source machine learning models and datasets found that code generation is the most common task and software development the dominant activity, while software management is entirely absent. That is a gap, and it points to where private data matters.
Public repositories show the final code and the public discussion around it. They rarely show how a company actually builds software: the internal chat where a design is argued out, the support ticket that became a bug, the planning documents, the call recordings with customers, the history of a proprietary codebase over several years. An AI lab training a coding agent or building an environment for agents to practice in needs exactly that texture, and it cannot be scraped.
The author of an R-based guide to software engineering data sources makes a related point: dataset selection overemphasizes size and attribute count, while provenance, versioning, the unit of analysis, missing-data policy, licensing and the anonymization of developer identities are often under-reported. Those same under-reported items are what a serious buyer asks you about, which we return to below.

Website behavior data: what public sets capture
The related query people search alongside this one is website behavior data. Public examples show what “behavior” means in practice:
| Dataset | What it captures | Scale |
|---|---|---|
| RUM Archive | Real-user-monitoring page loads and resource fetches, published daily under CC BY-SA 4.0 | About 200 million page loads and 160 million resource fetches per day |
| Retailrocket | Views, add-to-carts and transactions, with all values hashed | 2,756,101 events from 1,407,580 visitors over 4.5 months |
| OTTO sessions | Anonymized clicks, carts and orders in time order | 12 million sessions, 220 million events |
| WebWorldData | Web interaction trajectories: clicks, form fills, navigation, keyboard and mouse actions | 1.06 million trajectories |
Notice what these have in common. Identifiers are hashed or anonymized, time windows are short, and each covers one kind of event. A company that has logged its own product usage, onboarding flows, support interactions and purchase history over several years has something longer and richer, joined to context that a public sample cannot offer.
How people access public datasets
If you are researching how buyers work with data, the access routes matter, because they shape what buyers expect from a delivery.
The common formats are plain text, CSV, ARFF files, database dumps and remote APIs or web services, according to the same SE data guide. Sources range from open infrastructures to curated collections to restricted datasets.
Cloud-hosted public data is a major route. Google’s BigQuery public datasets are stored in BigQuery and offered through its public dataset program. Google pays for storage, you pay for queries, and the first 1 TB of query data processed each month is free. Access is through the console, the bq command-line tool or client libraries. Pricing for queries beyond the free tier is on Google’s BigQuery pricing page, which is where to check current rates, since they change.
Kaggle is the other familiar route. Its dataset documentation sets a 200 GB limit per dataset and encourages publishers to use open, non-proprietary formats such as CSV so that more people can work with the data.
For a company preparing to license data, the lesson is that buyers like clean, documented, well-structured deliveries. Exports from Slack, email systems, CRMs, ticketing tools, code hosting and call recording platforms are the normal starting points, and a buyer will want to know what is in them before it bids.

Licenses and privacy: why public terms are a poor template
Public datasets carry licenses that were written for research, and they show why license terms deserve attention in any commercial deal.
GHTorrent, a mirror of GitHub activity data, is a useful example. Its FAQ says it is dual-licensed: CC-BY-SA for non-commercial uses such as education, research and personal work, while commercial users must contact the maintainer. The same FAQ says it has not distributed personal information by default since March 2016 and that personal-data access has not been possible since May 2018. Users can opt out, after which their email is replaced and their real name removed.
The AWS Open Data registry is similarly plain about it: unless the documentation says otherwise, datasets are provided by third parties under various licenses, not by AWS.
The authors of the repository-mining directory put the principle well: data rights should be clearly described so that reuse is not limited by ambiguous licensing restrictions.
Translate that to your own data and the requirements are concrete:
- Know whose data is in your systems: employees, customers, contractors, third-party contributors.
- Know which contracts and terms of service restrict reuse.
- Decide what the buyer may do with the data, for how long, and whether the license is exclusive.
- Make sure personal information is removed or de-identified before anything leaves your environment.
Our seller checklist for AI data licensing agreements walks through the contract points to settle. For the question of where your data came from and how a buyer will verify it, see our piece on dataset provenance.
What to evaluate before you prepare your data
The SE data guide’s list of commonly under-reported attributes doubles as a readiness checklist for a seller. Here is how it maps to your situation.
Provenance. Which system produced each export, on what date, with what tool or query? A buyer pricing a Slack archive or a codebase history wants to know it is complete and unaltered.
Versioning and history depth. Longer history is more useful. The companies we work with typically have 20 to 500 staff and more than three years of records.
Unit of analysis. Is a record a message, a ticket, a commit, a call, a session? Buyers building agent-training environments think in terms of whole workflows, so say how records link together.
Data quality and gaps. Note missing periods, migrated systems and corrupted exports. A documented gap is a manageable detail. A gap a buyer discovers later is a problem for the license.
Rights and redistribution. Check customer contracts, employee agreements and third-party terms. Legal review belongs before any bidding, not after.
Privacy. Developer and customer identities need to be removed or masked. In our process, de-identification is done by an independent, owner-approved partner, and no company data moves until a deal is signed.
The kinds of data that tend to matter to AI buyers in this category are Slack and email, CRM records, support tickets, codebases, documents, call recordings and financial exports. Our guide to CRM data monetization for AI training covers one of those in depth, and the procurement data and supply chain datasets articles show how the same thinking applies in other functions.
Why one offer is rarely enough
Most companies that think about licensing their data get a single inbound approach, or go looking for one buyer. A lone buyer sets the price, the terms and the pace, and you have no reference point to judge the offer against.
A competitive process changes that. When several qualified buyers submit sealed bids for the same de-identified data, each has to put its best price and license terms forward without seeing the others. The seller compares price, term length, exclusivity and usage limits side by side. That is the logic behind our approach, and our stated range for company data is typically $100K to $4M, depending on what the data contains and how it is documented. Your own number depends on your data, which is why a valuation comes first. Our page on company data valuation for AI licensing explains what moves it.
For a view of the wider market, our AI data licensing market report is updated regularly.
How the process works with us
We act as a sell-side broker, which means we work for the data owner and not for the buyers. The steps look like this:
- Two-minute intake. You tell us what systems you have and roughly how much history.
- Free valuation. You see what your data may be worth before you commit to anything.
- Qualified buyers invited. Every qualified buyer is invited to bid, with no commission bias toward any particular one.
- Sealed bids and license terms. Bids come in with transparent license terms, and you keep control over those terms.
- De-identification and legal review. An independent, owner-approved partner de-identifies the data, and no company data moves until a deal is signed.
- Close. The winning buyer pays our commission. Sellers pay nothing upfront and nothing at all unless a deal closes.
Our security and data handling page sets out how information is protected throughout.
Our view
We keep seeing companies study public datasets to judge what their own data is worth, and the comparison misleads them. Public sets are built from what is free to collect. Your company’s data is the opposite: it is private, longitudinal and tied to real work.
The market already knows this. The gap in public data around software management, internal collaboration and real customer behavior is the reason private data gets bids at all.
Our job is to make those bids competitive. We represent the seller, invite every qualified buyer, keep your data where it is until a deal is signed, and let you compare sealed offers on price and on license terms. Sellers never pay us. We are paid by the winning buyer, only if a deal closes.
The risk we most want owners to avoid is underselling: taking the first offer because it was the only one.
Erik
Where to start
If your company has more than three years of engineering, support, CRM or behavior data, begin with a valuation. It takes a two-minute intake, costs nothing, and tells you whether the data is worth taking to bidders. Use the seller checklist alongside it to prepare your rights and contract questions.
Start with a free valuation and see how several AI data buyers would compete for your data.
FAQ
What are software engineering datasets used for?
Researchers use them to study defects, issue tracking, developer activity and how software evolves. AI developers use them to train and test coding models, for example with task sets built from real GitHub issues and pull requests. The same kinds of data from inside a company can be licensed to AI buyers.
Can I use public datasets commercially?
It depends on each dataset’s license. Some are free only for non-commercial use and ask commercial users to contact the maintainer, as GHTorrent does. Read the license on each dataset page before using it in a product.
Is website behavior data worth anything to AI buyers?
Buyers are interested in clicks, sessions, carts and navigation paths, as the public clickstream and web-interaction sets show. The value of your own logs depends on history depth, structure and how well personal identifiers are removed. A free valuation is the quickest way to get a figure for your data.
What does it cost to sell company data through Bright Info?
Nothing for sellers. There are no upfront fees, and we are paid a commission by the winning buyer only if a deal closes.
Is my data protected during the process?
Yes. De-identification is handled by an independent, owner-approved partner, and we do not move any company data until a deal is signed. License terms stay under your control.
Sources
- A Directory of Datasets for Mining Software Repositories
- A Versatile Dataset of Agile Open Source Software Projects
- From Bugs to Benchmarks: A Comprehensive Survey of Software Defect Datasets
- Towards a Classification of Open-Source ML Models and Datasets for Software Engineering
- nebius/SWE-rebench-V2 on Hugging Face
- tera-PROMISE repository
- Data Sources in Software Engineering
- RUM Archive datasets
- Retailrocket recommender system dataset
- OTTO Recommender Systems Dataset
- Qwen/WebWorldData on Hugging Face
- BigQuery public datasets
- BigQuery pricing
- Kaggle dataset documentation
- GHTorrent FAQ
- Registry of Open Data on AWS
Recommended
More from the blog
- Domain Adaptation Datasets: Benchmarks and Your Own Data
- Invoice Datasets: What They Contain and What They're Worth
- Predictive Maintenance Datasets: Where to Find Them and How to Choose
- All posts
Find out what your company's data is worth
Bright Info runs a sealed bid among AI data buyers for your record. You pay nothing unless a deal closes.
Get your data valued