OrderBy Laravel Eloquent Relationship

I would like to order listing_data based on listing_packages.order . Here is the query

ListingData::with(['listing_package'])->orderBy('listing_package.order','desc')->paginate(24);

I would like to order based on listing_package.order “DESCENDING”

However, it returns

SQLSTATE[42S22]: Column not found: 1054 Unknown column
‘listing_package.order’ in ‘order clause’

Here is the model ListingData.php

public function listing_package(): BelongsTo
{
    return $this->belongsTo(ListingPackage::class);
}

Here is the model for ListingPackage.php

public function listing_data(): HasMany {
    return $this->hasMany(ListingData::class);
}

Edit:

I also try this one

    $relations = [
        'district',
        'regency',
        'province',
        'category',
        'type',
        'rentType',
        'listing_package' => function ($q){
            $q->orderBy('order','DESC');
        },
    ];
    $listings = ListingData::with($relations)
    ->latest()
        ->paginate(24);

But the data is not sorted

Am I missing something here?