Check syslog, if there are lines:
... brcmfmac: brcmf_cfg80211_scan: scan error (-110)
... brcmfmac: brcmf_run_escan: error (-110)
Then it's buggy Broadcom drivers. Only option, for now, is to use different wifi chipset. e.g. dongle with Atheros chipset works fine