When Jeremiah finished telling the people all the words of the LORD their God—everything the LORD had sent him to tell them— Azariah son of Hoshaiah and Johanan son of Kareah and all the arrogant men said to Jeremiah, “You are lying! The LORD our God has not sent you to say, ‘You must not go to Egypt to settle there.’ But Baruch son of Neriah is inciting you against us to hand us over to the Babylonians, so they may kill us or carry us into exile to Babylon.”
So Johanan son of Kareah and all the army officers and all the people disobeyed the LORD’s command to stay in the land of Judah. Instead, Johanan son of Kareah and all the army officers led away all the remnant of Judah who had come back to live in the land of Judah from all the nations where they had been scattered. They also led away all the men, women and children and the king’s daughters whom Nebuzaradan commander of the imperial guard had left with Gedaliah son of Ahikam, the son of Shaphan, and Jeremiah the prophet and Baruch son of Neriah. So they entered Egypt in disobedience to the LORD and went as far as Tahpanhes.

If you notice, it said all the arrogant men rebuked Jeremiah, but all of the remnant followed them, not just the arrogant people. It seems people would start believing Jeremiah, because he's been right in everything he says, but that made no difference to the people, they listened to the arrogant men, and only a tiny remnant of people stayed behind.