True, for some farming elements are removed from the plot of land faster than natural replenishment. These will have to be added back by humans in some way. The main constituents of food, on the other hand, come from the atmosphere. Hydrogen, Carbon, Oxygen, and Nitrogen (with the right plants/bacteria). Soil is a mixture of organic matter, minerals, gases, liquids, and organisms that together support life [1]. This soil is needed for efficient agriculture and certain farming practices increase the amount of soil and other decrease it. Once it is gone, farming stops.

[1] https://en.wikipedia.org/wiki/Soil