Eloquent : condition based nested eager loading

I have 4 tables. combos, products, product_variants, combo_product (pivot table). combo_product has 2 columns viz reference, reference_id.

Now, I want to attach relationship condition based.

if(reference == 'product') then with('products.product_details') and

if(reference == 'product_variant') then with('products.product_variant_details').

This are the relationships I have:

Combo.php

public function products()
{
    return $this->hasMany('App\V1\Models\ComboProduct');
}

ComboProduct.php

public function product_details()
{
    return $this->belongsTo('App\V1\Models\Product', 'product_reference_id');
}
public function product_variant_details()
{
    return $this->belongsTo('App\V1\Models\ProductVariant', 'product_reference_id', 'product_variant_id');
}

I managed to get one relationship, but i can't get the other relationship condition based. Here is my code.

Combo::whereComboId($id)->with([
   'products' => function ($query) {
        $query->with(['product_details'])
            ->where('product_reference', 'product'); 
         }
    ])->get();

via Sankalp Tambe