May 5, 202610 min read

Pulling Public Instagram Profile and Post Data Without Scrapers

The requirement usually arrives sounding simple. A creator marketplace needs follower counts to rank profiles. An agency dashboard needs recent posts for fifty client accounts. A brand monitoring tool needs the comments on a campaign post.

All of that is public information — you can see it in a browser without logging in. Getting it into your database programmatically is where it stops being simple.

Why the official API usually is not the answer

Meta’s Instagram Graph API exists, and for one specific case it is the correct choice: reading data about accounts you or your users control, through an OAuth flow, for a Business or Creator account connected to a Facebook Page.

If that describes your product, use it. It is free, first-party, and stable.

It does not help when you need data about accounts that have no relationship with you — competitor profiles, a shortlist of creators you are evaluating, the hashtag your campaign is running under. There is no consent flow for “this person does not know my product exists”. That gap is why third-party data providers exist at all.

What running your own scraper actually costs

The build looks like a weekend. It is not, and the reason is that the maintenance is unbounded rather than the initial implementation being hard.

Blocking is the steady state. Instagram detects and blocks datacenter IPs aggressively. Working around that means residential proxies — an ongoing cost, an ongoing vendor relationship, and an ongoing rotation strategy.

The markup is not stable. Layouts and internal endpoints change without notice. Every change breaks your parser silently: the request succeeds, the selector matches nothing, and you write empty records until someone notices the dashboard is wrong.

The failure is silent. This is the part that catches teams out. A scraper that breaks loudly is annoying. A scraper that quietly returns partial data for three weeks corrupts your analytics and you cannot tell which rows are wrong.

Someone owns it forever. That is the real cost. Not the week to build it — the indefinite claim on an engineer’s attention, at unpredictable times, usually urgent.

Using a maintained API moves that whole category of work to somebody whose job it is. That, not the code, is what you are buying.

The endpoints

The Instagram API Collection covers profiles, posts, reels, media, comments, hashtag search, and two AI insight endpoints. Everything is a plain GET with query parameters:

curl --request GET \
  --url "https://instagram-api.p.rapidapi.com/v1/instagram/profile?username=nasawebb" \
  --header "X-RapidAPI-Key: $RAPIDAPI_KEY" \
  --header "X-RapidAPI-Host: instagram-api.p.rapidapi.com"
{
  "success": true,
  "data": {
    "id": "549313808",
    "username": "nasawebb",
    "full_name": "NASA Webb Telescope",
    "is_verified": true,
    "followers_count": 3547801,
    "following_count": 35,
    "media_count": 1302,
    "source": "apify",
    "fetched_at": "2026-07-20T10:10:24.746979+00:00"
  },
  "error": null,
  "meta": { "request_id": "req_example" }
}

Two fields in there deserve more attention than the follower count.

fetched_at is when the data was retrieved. Social metrics move constantly, and any answer you give is a snapshot. Store this alongside the values and surface it in your UI — “3.5M followers as of 20 July” is honest and defensible; an unqualified number implies a freshness you cannot guarantee.

source names the upstream provider. It is there so that when a number looks wrong, you can tell where it came from rather than guessing.

Posts, reels, and pagination

/v1/instagram/posts and /v1/instagram/reels take a username, a limit up to 100, and a cursor:

GET /v1/instagram/posts?username=nasawebb&limit=2

Use the cursor from one response to fetch the next page. Do not try to reconstruct pagination by incrementing an offset — cursor pagination exists because the underlying feed is not a stable indexed list, and offsets will duplicate and skip items.

Set limit to what you will actually use. Requesting 100 posts to display 6 is paying for 94 you throw away.

Media and comments

/v1/instagram/media takes a shortcode or a media_id; shortcodes are preferred. The shortcode is the segment in a post URL — instagram.com/p/Da52uFrle-Y/ → Da52uFrle-Y.

/v1/instagram/comments takes the same identifier plus a limit, which is what you want for campaign monitoring and sentiment work.

/v1/instagram/hashtag/search and /v1/instagram/search cover discovery — the “find creators posting about X” workflow that underpins most influencer tooling.

The AI insight endpoints

POST /v1/instagram/ai/profile-insights and POST /v1/instagram/ai/comment-insights are the two POSTs in the collection. They take profile or comment data and return a structured read of it, which saves you wiring up a model call yourself for the summarisation step of a creator report.

Designing around the constraints

Cache aggressively. Follower counts do not need re-fetching every page load. Cache profiles for hours, historical posts for much longer — a post from 2019 is not going to change. This is the difference between a viable unit economics and an unpleasant invoice.

Store snapshots, not current values. If you write followers_count into a single column and overwrite it, you have thrown away the growth trend, which is usually the thing your users actually want. Append rows with timestamps.

Expect gaps and design the UI for them. Accounts go private, get renamed, or get deleted. Reels are only returned where the provider exposes reel media, so a profile can legitimately come back with fewer than you expected. Your interface needs a state for “no data” that is not a spinner forever or a crash.

Handle errors as a normal path. Every response uses the same { success, data, error, meta } envelope, so one handler covers the collection. Watch for RATE_LIMIT_EXCEEDED (429) and back off — see Errors and Rate limits.

The part that is your responsibility

An API removes the engineering problem. It does not remove the legal and ethical one, and that distinction matters.

Public does not mean unrestricted. Data being visible without a login does not by itself make every downstream use lawful. Instagram’s terms, and the rules that apply to you, still apply.

Personal data is regulated wherever the person is. A public Instagram profile is personal data under GDPR and comparable regimes. If you store it and you serve EU or UK users, you need a lawful basis, a retention policy, and an answer for deletion requests. “It was public” is not a lawful basis on its own.

Aggregate analysis and profiling are different activities. Counting posts under a hashtag to measure campaign reach sits very differently from building individual dossiers on named people.

Delete when the source deletes. If an account goes private or a post is removed, that is a signal. Continuing to serve a cached copy is the behaviour that generates complaints, and it is easy to avoid.

Get advice appropriate to your jurisdiction and your use case. This is not it.

For the other half of social listening, see Twitter/X data for developers — the same category, a rather different data model, and its own set of things worth knowing before you build on it.