The conceptual design of the BRIKEN neutron detector at the radioactive ion beam factory (RIBF) of the RIKEN Nishina Center is reported. The BRIKEN setup is a complex system aimed at detecting heavy-ion implants, β particles, γ rays and β-delayed neutrons. The whole setup includes the Advanced Implantation Detection Array (AIDA), two HPGe Clover detectors and up to 166 3He-filled counters embedded in a high-density polyethylene moderator. The design is quite complex due to the large number and different types of 3He-tubes involved and the additional constraints introduced by the ancillary detectors for charged particles and γ rays. This article reports on a novel methodology developed for the conceptual design and optimisation of the 3He-counter array, aiming for the best possible performance in terms of neutron detection. The algorithm is based on a geometric representation of two selected detector parameters of merit, namely, the average neutron detection efficiency and the efficiency flatness as a function of a reduced number of geometric variables. The response of the neutron detector is obtained from a systematic Monte Carlo simulation implemented in GEANT4. The robustness of the algorithm allowed us to design a versatile detection system, which operated in hybrid mode includes the full neutron counter and two clover detectors for high-precision gamma spectroscopy. In addition, the system can be reconfigured into a compact mode by removing the clover detectors and re-arranging the 3He tubes in order to maximize the neutron detection performance. Both operation modes shows a rather flat and high average efficiency. In summary, we have designed a system which shows an average efficiency for hybrid mode (3He tubes + clovers) of 68.6% and 64% for neutron energies up to 1 and 5 MeV, respectively. For compact mode (only 3He tubes), the average efficiency is 75.7% and 71% for neutron energies up to 1 and 5 MeV, respectively. The performance of the BRIKEN detection system has been also quantified by means of Monte Carlo simulations with different neutron energy distributions.