Laravel/Eloquent query too slow

I have a model Person.php with a one to many relationship to Pet.php. I need to sort persons by the count of pets the person has. This query works:

$persons = Person::join('pets', 'persons.id', 'pets.person_id')
    ->select('persons.*', DB::raw('count(pets.id) as pets_count'))
    ->groupBy('id')
    ->orderBy('pets_count', 'desc')
    ->limit(100)
    ->get();

SQL:

select `persons`.*, count(pets.id) as pets_count 
from `persons` 
inner join `pets` on `persons`.`id` = `pets`.`person_id` 
group by `id` 
order by `pets_count` desc 
limit 100

The database is seeded with 50,000 persons and 100,000 pets. Sorting by pets_count takes 350ms (while sorting by a persons column without join takes 18ms). I haven't added any indexes.

How can I make this faster?

I could add pets_count column to persons, and have to update it often and carefully, and then this query could be sped up without the joins and counts. Is this good practice? Does SO have a votes_count column on questions, or do they re-count the votes often?

via billyhafiz