On this page
Key takeaways
A professional profile dataset is one record per person: identity, location, current and past roles, education, skills and certifications, with the date each record was last checked. It is licensed annually and delivered as compressed files to your S3 or GCS bucket, or as a Snowflake private listing.
- Every record carries
date_updated(the last crawl) andschema_version(the shape of the payload). Read both before you load a row. - Delivery cadence (daily, weekly, monthly) is a separate setting from how often records are recrawled.
- Display, resale and retention terms are contract questions. Ask them before the sample run, not after.
What the search turns up
Search for a profile dataset and page 1 is two Reddit r/datasets threads, a Kaggle notebook, a 2022 post about a dataset product that no longer exists, a vendor's GitHub sample, a ResearchGate figure and a jobs scraper. Three different jobs share one results page.
A research sample is a fixed file from one point in time. The fields are whatever the author kept, and the license is rarely stated. The GitHub sample at #4 lists company fields only, gives two different record counts on the same page and says nothing about terms (retrieved 2026-11-01). Fine for a notebook. Not something you ship on.
A scraped file you build yourself gives you the fields you choose and the maintenance that comes with them: breakage when the page changes, blocking, and the legal exposure of the collection method.
A licensed feed has a published schema, a delivery schedule, written terms and records that get recrawled over time. That is the one a product team can build on, and it is the one this article covers, using Forager's published person schema as the worked example.
person and company records in one dataset.
Anatomy of one record
The person schema (version 2.0.4) has 35 top-level fields. Two are required: forager_id and date_updated. Everything else is optional, so a loader has to accept nulls and empty arrays on every other field. Grouped by what they are for:
| Group | Fields | Notes |
|---|---|---|
| Identity | forager_id, linkedin_id, linkedin_url, linkedin_slug, first_name, last_name, headline, description, photo | forager_id is the stable key. A small subset of records has no profile identifier, usually because the profile was banned. |
| Location | country, area, city, location, locations[], linkedin_country, linkedin_area | Each entry in locations[] has a name and a place_type. |
| Roles | roles[] | Each role: role_title, start_date, end_date, employment_type, organization_id, organization_name, organization_domain, organization_industry, location. |
| Education | educations[], courses[], test_scores[] | school_name is required on an education entry; degree, field_of_study, grade and dates are optional. |
| Credentials | certifications[], languages[], honors[], patents[], publications[], projects[] | A certification carries certificate_name (required), authority, certificate_id, certificate_url and dates. |
| Profile signals | skills[], industry, number_of_connections, number_of_followers, is_available, volunteering[], volunteer_causes[], organizations[] | skills is a plain array of strings. |
| Freshness and version | date_updated (in the record) plus schema_version (on the row) | Covered in the next two sections. |
Two details in roles[] matter for anyone building "current employer":
- There is no
is_currentflag. A role with noend_dateis the current one. A person can have more than one. employment_typeis a numeric code, not a label.12isfull_time,2iscontract,18isinternship; the full code table is on the Role schema page.
The full dictionary, with every sub-object, is at data-license/v2/schema.
How it arrives
Each row in a delivered file is a wrapper around the record:
| Field | What it holds |
|---|---|
id | Forager record id (integer) |
date_updated | When the record was last crawled, UTC |
schema_version | Version of the payload, for example 2.0.4 |
serialized_data | The record itself, shaped by the schema for that feed type and version |
Files in a bucket. NDJSON is the default; CSV, TSV (any delimiter) and Parquet are also supported. Files are UTF-8, partitioned and compressed with gzip, bz2, brotli or zstd. The path tells you what is inside before you open it:
/forager_data_feed/<data_feed_id>/<date_created>/<data_feed_type>_<schema_version>/data_<partition_id>.<file_type>.<compression_type>
/forager_data_feed/123/2025-10-15/person_2.0.4/data_0_0_1.json.gzip
data_feed_type is person, organization or job. The schema version sits in the directory name, so a version change shows up in the path before it shows up in a row.
Drops go to an Amazon S3 bucket you own (the preferred route: create the bucket, attach the documented IAM policy to a user for Forager, send the keys) or to Google Cloud Storage. Azure Blob is coming soon. Cadence is daily, weekly or monthly.
Snowflake private listing. Request the Forager.ai listing for each feed in your license on Snowflake Marketplace, then pick it up under Data Products, Private Sharing, with a role that has CREATE DATABASE and IMPORT SHARE. The feed appears as a database you query like any internal table.
Net-new transfers. A transfer can be limited to records crawled since the last scheduled transfer, so a weekly drop is the week's recrawls rather than the whole dataset again. Ask which mode your feed uses.
Docs: output file details, S3, GCS, Snowflake.
Reading freshness
date_updated is the moment the record was last crawled. It is the only freshness fact in the file, so three things follow from it.
The total count and the updated count differ. A feed's total is every record currently in the dataset. The "updated in the last month" count is how many of those were recrawled in that window. The gap is expected. Records stop being crawled when a profile goes inactive, is banned or no longer exists, and a record that is not crawled is not transferred again. So the records that stop moving are the ones that stopped existing.
Delivery cadence is not refresh cadence. Daily, weekly or monthly controls how often recrawled records reach you. It says nothing about how often the underlying data is recrawled. Forager's published line is that records are recrawled on a rolling basis; the refresh term belongs in your agreement, in writing. Any vendor's cadence claim should be a contract term, and one you can check against date_updated yourself.
Measure it on your side. Plot the distribution of date_updated across the records your product cares about, then plot it again next month. That curve is the refresh rate as it applies to you, not as it applies to the whole dataset. B2B data decay: how to measure it on your own list walks through the method.
Joining the three feeds
A license can cover three feeds: person, organization and job post. They join on organization keys.
- A person's
roles[]entry carriesorganization_id,organization_nameandorganization_domain. - An organization record carries
forager_id,domain,name,industry,employees_range,countryandcity. - A job post carries
organization_forager_idandorganization_domain, plustitle,url,date_featured,is_activeandis_remote.
The documented recipe flattens roles, keeps the ones with no end_date, lowercases the domain and joins it to the organization feed:
CREATE OR REPLACE VIEW CURRENT_ROLES AS
SELECT
p.serialized_data:forager_id::INTEGER AS person_id,
r.value:role_title::STRING AS role_title,
LOWER(r.value:organization_domain::STRING) AS organization_domain
FROM PERSON_FEED p,
LATERAL FLATTEN(input => p.serialized_data:roles) r
WHERE r.value:end_date IS NULL
AND r.value:organization_domain IS NOT NULL;
Then JOIN ORGANIZATION_FEED o ON LOWER(o.serialized_data:domain::STRING) = organization_domain. To add job posts, join organization_forager_id to the organization's forager_id. Watch for people with two open roles (they appear twice), organizations that share a domain, and roles with no domain at all (count them so you know how many drop out). On job posts date_updated can be null; on person and organization records it is always set.
Full recipe: Join the people and company feeds in Snowflake.
License questions to ask any vendor
A sample file tells you what the data looks like. The agreement tells you what you may do with it. Six questions worth asking before the sample run, with Forager's answers where they are published:
- Can my users see it? Whether records, or fields from them, can be displayed inside your product to your own customers. Forager: agreed in the license agreement.
- Can I pass it on? Whether you may resell, sublicense or redistribute records or derived datasets. Forager: agreed in the license agreement.
- What happens at the end of the term? Whether delivered copies must be deleted, and whether models or indexes trained on them count. Forager: set in the license agreement; ask before the sample run.
- How are removals handled? What the vendor does when a record is no longer valid or legally distributable, and what you are expected to do with copies already delivered. Forager stops crawling the record and it is not transferred again.
- Whole dataset or a subset? Whether the license can be scoped to the regions, industries or roles you need. Forager: annual license, the entire dataset or a custom subset.
- How do schema changes reach me? Forager versions the schema; a new version is released when fields are added or structures change, changes are designed to be additive wherever possible, and the version is in every file path and every row, so you migrate on your own timeline.
Evaluate with a sample on your own universe
Record counts describe the dataset. Fill rate describes the dataset where you need it. Measure the second one yourself:
- Take the people your product already knows about, not a list the vendor picks: a few thousand
linkedin_slugorlinkedin_urlvalues from your own tables. - Match them against the sample and count the hits.
- For the hits, count fill per field: a current role (
end_datenull), a non-emptyeducations[], askills[]with more than a few entries, acity. - Look at the
date_updateddistribution for those records, not for the sample as a whole. - Repeat on a second sample a month later if you can. The change tells you more than either snapshot.
The numbers that come out are yours. We publish no fill rate, because the only one that matters is the one on your records.
FAQ
What fields are in a person record?
Identity (forager_id, network ID, profile URL and slug, names, headline, description, photo), location (country, area, city, location, locations[]), roles[], educations[], certifications[], skills[], languages[], projects[], honors[], patents[], publications[], courses[], test_scores[], volunteering[], organizations[], follower and connection counts, industry, is_available and date_updated. Only forager_id and date_updated are required.
How is the dataset delivered?
As compressed files (NDJSON by default; CSV, TSV or Parquet) dropped daily, weekly or monthly into your own S3 or GCS bucket, or as a Snowflake private listing you query in place. Azure Blob is coming soon.
What does date_updated mean?
The time the record was last crawled, in UTC. It appears on the row wrapper and inside the record. Records that are no longer crawled keep their last value and are not transferred again.
Can I receive only new or changed records?
Yes. A transfer can be limited to records crawled since the last scheduled transfer.
Can I display or resell the data?
Both are license terms, agreed per contract. Ask for them in writing before you build on a sample.
Bring the identifiers you already hold
Person, company and job-post feeds across 800M+ people and 65M+ companies, delivered to your warehouse on an annual license. Bring the identifiers you already hold and we will run the sample.
- 800M+ people and 65M+ companies
- Mobile and email lookups that find nothing cost 0 credits
- Free API key to start
#1 of 10 in Clay's 2026 test of mobile data providers Source: Clay