Doing nothing on your end does not make it on server end. Could be that others on your internet node are also using bandwidth, your isp sees a raise in your usage and are throttling the connection, your Fair Access Policy FAP is kicking in, or internet problems.
I've noticed throttling from my dsl- worse during daylight and early evening. speedtest your connection a few times a day, and maybe run one while streaming too. Streaming doesn't use most people's internet to the max and a speedtest might show they're throttling