Laravel/Eloquent query too slow

I have a model Person.php with a one to many relationship to Pet.php. I need to sort persons by the count of pets the person has. This query works:

$persons = Person::join('pets', 'persons.id', 'pets.person_id')
    ->select('persons.*', DB::raw('count(pets.id) as pets_count'))
    ->groupBy('id')
    ->orderBy('pets_count', 'desc')
    ->limit(100)
    ->get();

The database is seeded with 100,000 persons and 200,000 pets. Sorting by pets_count takes about 500ms longer than other queries. Am I doing something wrong?

via billyhafiz