The conventional iterative methods are the most common approaches used to optimal design of double-pipe heat exchanger (DPHE) systems. These methods employ a simple iterative process applying an initial guess to estimate a sequence of enhancing approximate solutions. Such a working mechanism makes them very sensitive to the initial conditions of the optimization process. Also, the proper step size(s) definition is a very important factor for avoiding these algorithms' divergence. Furthermore, in most of these techniques, the regular approaches (e.g., penalty function) are applied to control the constraints of the problem. To overcome these drawbacks, in the current study, instead of the mentioned conventional approaches, a recently developed nondeterministic method, so-called interactive search algorithm (ISA), is integrated with the improved fly-back (IFB) mechanism to provide a robust optimization algorithm for solving the DPHE's thermo-economic optimization problem. The IFB technique is an efficient strategy for handling the constraints of the problem. This mechanism ensures the feasibility of the achieved optimal solutions by excluding any violated candidates. Consequently, the search capability of the compound ISA-IFB technique is tested on a suite of mathematical functions and the thermo-economic model of a DPHE system. For more clarity, the attained optimal solutions using the presented method are compared with five other well-established optimization techniques. Acquired outcomes indicate that the ISA-IFB method in optimizing both mathematical functions and DPHE problem gives promising results in terms of accuracy, stability, and convergence rate.