If you want facts, then, the facts are that a meteoric rock crashed near the present location of Mecca which was the oasis of the bani al Quraish Jewish Tribe.
This was taken for the Sign of God and a Cult was built of the God - al Lah in Arabic, who had three daughters - al Lat, al Uazza and al Mannat. The rock, which was in pieces, was gathered and put into a vagina shaped container [in honor of the Goddesses] and called the al Hajarul Aswad or the Holy Stone.
A Pilgrimage was created and made mandatory to all Cult Members called the Haj and the then rectangular shaped building called the Kaaba, aka the Ba'it' ullah or the House of al Lah or Allah, was then built by the bani al Quraish faction that left Judaism to embrace the new Religion. The initial purpose of the Kaaba was to house the Kaaba Stone and the 300 and odd Statues of the God - al Lah and his daughters in various combination.
Mohammad ibn Abdullah al Muttailib al Quraish, or Mohammad, son of Abdullah [Slave of al Lah], of the town of Mutta [not too far from Mecca], of the Tribe of the Quraish, was born a bani al Quraish Jew, inheriting his Judaism from his father, and got his Pagan part of the Cult from his mother - Amina binti Wahab. His first wife - Khadija binti Khuwaylid was also a Cult Member of the people of the Kaaba.
After Mohammad began his New Cult of Islam which was, in the beginning, a Christian Cult, he destroyed all the statues in the Kaaba after taking for his New Cult.
According to many Arab Historical records, including Muslim ones, all these facts are confirmed and it is also recorded that the Kaaba was destroyed 5 times in it's existence, so far.
The first two were because of natural Calamities, earthquakes to be precise, and the third happened in Mohammad's time when a flash flood suddenly destroyed it a little after he destroyed all the statues there and he also helped in it's being rebuilt.
The last two times that it was destroyed were by the Muslims, themselves. The first was by the Ummayad Caliph Yasid ibn Muwaiyya and his General Abdul Malik and the last was by the Khwajirites who destroyed it and took the Kaaba Stone - al Hajarul Aswad and ransomed it for a heavy price to the Abbassids who then restored the Kaaba and the stone as we now see it.
These are facts based upon the various Arab Historical Texts and the Sahih or Authentic books of the Muslims like the Sirat rasool'illah, the Tafsir ibn Kathir and the Sahih Hadiths.
Please refer the History of Arabia from the various texts for more details.
Present day Muslims, however, have a different version in which they claim it was built by a 90 foot tall Adam.
It's present purpose, however, is the Qiblah or the Direction of the Prayers; which Mohammad changed from the original position which was earlier Jerusalem.
.