Besides the removal of (in islam) addition.
Pay attention to another verse within the same chapter..
To every nation We appointed commandment/acts/rites/tasks (Mansakh) which they perform, therefore do not dispute with them about the matter and invite them to your Rabb (read: and not invite them to your religion or sect); most surely you are on a right way.
And based on those 'instructions', Muhammad decided to allow the Meccan Arabs to keep their tradition of Jogging around in cubicle black stone.. yet he changed its meaning... and cleansed it from Idols.. and other superstitions (as much as he could)...
Salam / Peace
Exactly. This is why when people say haj is a pagan ritual, its on so if its gods and idols they are doing it for. Ritauls are common forms of worship, the question here is to whom the worship is directed to.
Since Haj was always a Meccan form of ritual, there was always going to be the problem that other cities will create their own haj, after all people pray in different cities and fast in different cities so why all do haj in Mecca only? This is why the Sunni authorities led by the Meccan dynasty had to make Haj in Mecca an obligation and one of the pillars of Islam and created many stories like Adam or Abraham or Hagar as being there to give it significance and exclusivity. We know that during the Ummayid dynasty a rival Kaba was created.
Not that Abraham was there or not, he could have bene there or not. But many prophets never had haj in Mecca like Moses, like Jesus or Lot or Job or Saleh. So how can it be an obligation? We can say its a Muhamedan tradition but that does not make it an obligation.
But you are absolutely correct, as long as the Meccans perform these rituals to God and not the Meccan idols, then its a valid form of worship.