Whatever connection they had with men was lost to time. The Rings of Power leaned into that aspect of Hobbit history when Nori and Poppy encountered a Stoor named Merimac in the desert of Rhûn.
The show introduced Smials as ground holes or mainly made of dry grass and straw which also matches the description of primitive Hobbit holes as reported by Screenrant, "thatched with dry grass or ...