To integrate the area under a curve you would use infinitesimals.

No this is not true. Each rectangle in a Riemann sum has nonzero base. There are no infinitesimals in the real numbers.

Perhaps I should start by defining infinitesimal. An infinitesimal is a number that is greater than $$0$$ but less than $$\frac{1}{n}$$ for any positive integer $$n$$.

It's clear that there are no infinitesimals in the real numbers. If you claim $$\epsilon$$ is infinitesimal, I'll just keep going $$.1, .01, .001, .0001, \dots$$ till I find an $$n$$ such that $$0 < \frac{1}{n} < \epsilon$$, falsifying the claim that $$\epsilon$$ is infinitimal.

When we form a Riemann sum, we partition the $$x$$-axis into a finite collection of subintervals, each one finite, and form the sum. If the collection of all such sums has a limit, we call that sum the value of the integral.

No infinitesimals are involved in this process, nor could they be, since there are no infinitesimals in the real numbers.

The notation $$dx$$ is not an infinitesimal. It is simply a notation telling you which variable you're integrating with respect to. In more advanced math such as differential geometry, they give formal meaning to $$dx$$ and other differential forms, as they're called, but they are not infinitesimals.

It's true that many students come away from calculus class thinking that $$dx$$ is an infinitesimal, but it is not. The historical achievement of making calculus logically rigorous involved banishing infinitesimals from math and replacing them with limits. Instead of talking about "infinitely small" we talk about "arbitrarily small" and this makes all the difference.

It's true that there are alternative number systems such as the hyperreals and the surreals that contain infinitesimals, but these systems add no clarity to calculus and have their own logical difficulties, such as being incomplete and requiring more set theoretic assumptions than do the standard reals. They're interesting to study in their own right, but they do not bear on the question of whether there are infinitesimals in the real numbers.

There are no infinitesimals in the real numbers.