TMDb Discover Returning Inconsistent ID Counts when no Sorting is defined

Поддръжка на The Movie Database

публикувана от yozoraxcii
STAFFMOD
на 21 септември 2024 в 6:44 AM

We have stumbled across an issue when using TMDb Discover to fetch IDs.

When a sort option is not defined and the default sorting is used, the amount of IDs that is returned varies wildly (hundreds of items different).

To demonstrate this we have set up a script which:

Defines an API key (apikey) for accessing the API.
Sets the number of runs (number_of_runs) and the delay between runs (delay_in_seconds).
Makes a request to the TMDb API to discover movies associated with a specific company (company ID 33).
Retrieves the total number of pages and results from the API response.
Iterates through all pages to collect movie IDs.
Logs the results of each run, including the total pages, total results, and the list of movie IDs.
Stores unique movie IDs and compares them across runs to identify differences.
Logs unique IDs found in the current run that were not present in previous runs and vice versa.

The python script is available here, simply replace APIKEY with a valid API Key

For this test, I set the runs to be 30 seconds apart with 10 total runs.

My first run's parameters are this, with no sorting defined:

    url_params = {
        "api_key": f"{apikey}",
        "with_companies": "33",
        "language": "en",
        "page": 1
    }

The log file can be found here: GitHub Gist - No Sorting

As can be seen in the log, the ID Count fluctuates from 2614, to 2129, back up to 2614, down to 2129 etc.

The log also shows that there are IDs in subsequent runs that do not appear in the first run, and vice-versa.

My second run's parameters are identical, apart from adding sorting:

    url_params = {
        "api_key": f"{apikey}",
        "with_companies": "33",
        "language": "en", 
        "sort_by": "primary_release_date.asc",
        "page": 1
    }

The log file can be found here: GitHub Gist - Primary Release Date Sorting

As can be seen in the log, the ID Count is consistently set to 2614 and the results never feature IDs that have not appeared before - it is consistently the same 2614 IDs.

11 отговора (на страница 1 от общо 1)

Отговор от yozoraxcii

STAFFMOD

на 21 септември 2024 в 2:08 PM

A couple other things I have tried to no success:

Specifying "sort_by": "popularity.desc"
Adding a 1 second sleep between each page load to see if perhaps a rate limit was being hit

In both of these scernarios, the amount of unique IDs differed from run-to-run, meaning that the same ID is appearing on more than one page of the response only when the sort order is popularity (which I assume to be the default sort order if it is not specified).

EDIT:

Interestingly, popularity.asc works with no issues, this seems to be limited to popularity.desc

t🇧

Отговор от ticao2 🇧🇷 pt-BR

STAFFMOD

на 21 септември 2024 в 3:58 PM

I'm not an expert.
If I understand your question correctly, then...

If you don't specify the Sort, the default &sort_by=popularity.desc will be used.

sort_by - string - Defaults to popularity.desc

Popularity and Vote are values that change constantly throughout the day.
Read here:
https://developer.themoviedb.org/docs/popularity-and-trending
That's why it's normal for a movie that was on page 3 to move to page 2.
But, we have the problem of page CACHE.
By definition, the CACHE has a lifetime.
The CACHE is not cleared instantly or simultaneously with changes in Popularity, Vote or Trend values.
The CACHE is not cleared instantly, it is simultaneous with the changes in Popularity, Vote or Trend values.
That's why this "error" of the same movie being on page 2 and 3.

I don't know if this helps you.

EDIT
I believe this explanation also applies to the use of the Trending Request.

t🇧

Отговор от ticao2 🇧🇷 pt-BR

STAFFMOD

на 21 септември 2024 в 4:05 PM

Regarding the different behavior between "popularity.asc" and "popularity.desc".
I believe that the less popular ones are less seen and/or their values change throughout the day.
So if the answer starts with the less popular ones it will probably be a more stable answer, with no issues.

Отговор от yozoraxcii

STAFFMOD

на 21 септември 2024 в 4:08 PM

@ticao2 said:

I'm not an expert.
If I understand your question correctly, then...

If you don't specify the Sort, the default &sort_by=popularity.desc will be used.
sort_by - string - Defaults to popularity.desc
Popularity and Vote are values that change constantly throughout the day.
Read here:
https://developer.themoviedb.org/docs/popularity-and-trending
That's why it's normal for a movie that was on page 3 to move to page 2.
But, we have the problem of page CACHE.
By definition, the CACHE has a lifetime.
The CACHE is not cleared instantly or simultaneously with changes in Popularity, Vote or Trend values.
The CACHE is not cleared instantly, it is simultaneous with the changes in Popularity, Vote or Trend values.
That's why this "error" of the same movie being on page 2 and 3.

I don't know if this helps you.

EDIT
I believe this explanation also applies to the use of the Trending Request.

The issue isn't just duplicate entries - it's also that items that are returned from run 1 are completely absent from run 2 happening 30 seconds later, and can then reappear in run 3 30 seconds after that.

Cache is probably a good avenue to explore further, this doesn't seem to be intended behaviour though - if you and I both run a script to fetch movies from a company we should get the same results.

t🇧

Отговор от ticao2 🇧🇷 pt-BR

STAFFMOD

на 21 септември 2024 в 4:13 PM

@yozoraxcii said:

The issue isn't just duplicate entries - it's also that items that are returned from run 1 are completely absent from run 2 happening 30 seconds later, and can then reappear in run 3 30 seconds after that.

Cache is probably a good avenue to explore further, this doesn't seem to be intended behaviour though - if you and I both run a script to fetch movies from a company we should get the same results.

Well, as I said at the beginning, I'm not an expert.
Let's wait for administrator Travis Bell to stop by.

Отговор от yozoraxcii

STAFFMOD

на 21 септември 2024 в 4:19 PM

Just to add - After running more tests, this doesn't appear to impact tv discover (from what I've seen) and also doesn't impact companies with less results (company ID 1632 has 24 pages and doesn't seem to have this issue, compared to ID 33 which has 130 pages and has this issue).

t🇧

Отговор от ticao2 🇧🇷 pt-BR

STAFFMOD

на 21 септември 2024 в 4:24 PM

In any case, I think it would be good to know which API Request is having problems.
Not your code that builds the request, but the Request that is sent to our server.
Probably something like:

https://api.themoviedb.org/3/discover/movie?page=1&sort_by=popularity.desc&with_companies=33
https://api.themoviedb.org/3/discover/movie?api_key=THE_KEY&page=1&sort_by=popularity.desc&with_companies=33

Отговор от yozoraxcii

STAFFMOD

на 22 септември 2024 в 7:13 AM

@ticao2 said:

In any case, I think it would be good to know which API Request is having problems.
Not your code that builds the request, but the Request that is sent to our server.
Probably something like:
https://api.themoviedb.org/3/discover/movie?page=1&sort_by=popularity.desc&with_companies=33
https://api.themoviedb.org/3/discover/movie?api_key=THE_KEY&page=1&sort_by=popularity.desc&with_companies=33

Sorry for delayed response - the queries being sent are visible in the logs

https://api.themoviedb.org:443 "GET /3/discover/movie?api_key=MYAPIKEY&with_companies=33&language=en&page=1

Отговор от superboy97

STAFFMOD

на 22 септември 2024 в 7:51 AM

Correct languages are of the form en-US. With just en, the result is not guaranted.

Отговор от yozoraxcii

STAFFMOD

на 22 септември 2024 в 8:13 AM

@superboy97 said:

Correct languages are of the form en-US. With just en, the result is not guaranted.

Good spot - I'll make that amendment and run some more tests.

Отговор от yozoraxcii

STAFFMOD

на 22 септември 2024 в 11:58 AM

@superboy97 said:

Correct languages are of the form en-US. With just en, the result is not guaranted.

I have made this amendment and can confirm the issue still exists - items are only appearing in some runs.

URL is https://api.themoviedb.org:443 "GET /3/discover/movie?api_key=MYAPIKEY&with_companies=33&language=en-US&page=1

Log file is available here: TMDb Log with language set to en-US

I'll also highlight that a lot of these runs have approximately 500 duplicate IDs (approximately 20% of the result set).

Here is a popularity.asc log which does not have any of these issues

And here is one where I've specifically set popularity.descrather than relying on a "fallback" sort

Участници в дискусията

Категории

Поддръжка на The Movie Database