Every building upgrade gives you certain number of points (the longer it takes to upgrade a building, the more points you get).

Your village points are based ONLY on buildings you have at the village. Points are used to ascertain the 'size' of neighboring villages however, points alone cannot tell you exactly what buildings are in this village.