If the end result is that psychedelics get used as an excuse to take power away from the FDA, then everybody’s safety gets compromised in all areas of healthcare.

Yes, but it shouldn’t be legalized for the wrong reasons. We used to justify legalization using arguments about personal freedom for recreational use and pushing for more rigorous research into the therapeutic use cases. Now its popularity in the population is just used to push a pseudo-scientific and anti-science agenda.