To study the nonlinear behaviour of the X-ray binaries and distinguish between the chaotic and stochastic nature of their emission, we propose a novel method, which is based on recurrence analysis of time series. Widely known in other fields of physics, this powerful method is used for the first time in an astrophysical context. We analyze the X-ray lightcurves of several microquasars during their outburst and search for traces of chaotic behaviour, in particular, we estimate the indicators of deterministic chaos quantitatively, such as the Rényi's entropy for the observed time series, and we compare them with surrogate data. We define the NLD indicator as the averaged result of this procedure for a range of parameters of the recurrence analysis. We find strong indications of non-linear processes in some of the temporal states of the well-known microquasar GRS 1915+105, as well as its recently discovered analogue, IGR J17091-3624. Moreover, from our selected set of observations of other sources, also GRO J1655-40, GX 339-4 and XTE J1550-564 show considerably high NLD in some states. For the latter source, we provide a detailed study of temporal and energy evolution of NLD during its 1998/1999 outburst. We discuss the non-linear dynamics of the accretion flow in the context of the disc-corona geometry and propagating oscillations, possibly connected with the shocks in low angular momentum component of the flow, or other scenarios such as the radiation pressure instability model.