This assumption -- like many other assumptions about the Book of Mormon -- comes from a naive reading of the text that was filtered through the 19th century misunderstanding of the human migrations that populated the ancient New World.
Early American settlers were fascinated with the fact that the New World was already inhabited by indigenous people. From where did these people originate? A number of frontiersman theorized that the Indians were remnants of the ten lost tribes of Israel. At first blush, this theory seemed to fit fairly well with the overall story of the Book of Mormon, however, the Book of Mormon peoples did not purport to come from any of the "lost tribes."